[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:39:22 dist-test-slave-x4sn krb5kdc[16754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:39:22 dist-test-slave-x4sn krb5kdc[16754](info): set up 2 sockets
Sep 26 14:39:22 dist-test-slave-x4sn krb5kdc[16754](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 14:39:23.421427 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:39:23 dist-test-slave-x4sn krb5kdc[16754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:39:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:39:23.561362 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:40107
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:40107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:23.676988 16767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:23.677264 16767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:23.677305 16767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:23.681298 16767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:23.681355 16767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:23.681372 16767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:23.681387 16767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:23.681401 16767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:23.687036 16767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:40107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:40107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:23.688342 16767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:23.689466 16767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:23.699448 16775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:23.699520 16767 server_base.cc:1034] running on GCE node
W20260926 14:39:23.699466 16773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:23.701764 16772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:23.702282 16767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:23.703649 16767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:23.704833 16767 hybrid_clock.cc:648] HybridClock initialized: now 1790433563704779 us; error 75 us; skew 500 ppm
Sep 26 14:39:23 dist-test-slave-x4sn krb5kdc[16754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433563, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:23.713508 16767 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:39:23.714942 16767 webserver.cc:460] Webserver started at http://127.16.90.254:39533/ using document root <none> and password file <none>
I20260926 14:39:23.716277 16767 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:23.716369 16767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:23.716889 16767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:23.720120 16767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "d86c62766fc648b0a00db9f59723f86a"
format_stamp: "Formatted at 2026-09-26 14:39:23 on dist-test-slave-x4sn"
server_key: "47aa7d6e174a772bf4e58c229a93dbfd"
server_key_iv: "030d57f3239f24becec4f704eb8a9b8c"
server_key_version: "encryptionkey@0"
I20260926 14:39:23.721318 16767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "d86c62766fc648b0a00db9f59723f86a"
format_stamp: "Formatted at 2026-09-26 14:39:23 on dist-test-slave-x4sn"
server_key: "47aa7d6e174a772bf4e58c229a93dbfd"
server_key_iv: "030d57f3239f24becec4f704eb8a9b8c"
server_key_version: "encryptionkey@0"
I20260926 14:39:23.726910 16767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 14:39:23.730926 16782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:23.732940 16767 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:39:23.733064 16767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "d86c62766fc648b0a00db9f59723f86a"
format_stamp: "Formatted at 2026-09-26 14:39:23 on dist-test-slave-x4sn"
server_key: "47aa7d6e174a772bf4e58c229a93dbfd"
server_key_iv: "030d57f3239f24becec4f704eb8a9b8c"
server_key_version: "encryptionkey@0"
I20260926 14:39:23.733192 16767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:23.765767 16767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:23.768919 16767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:23.769151 16767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:23.777191 16767 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:40107
I20260926 14:39:23.777226 16834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:40107 every 8 connection(s)
I20260926 14:39:23.778371 16767 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:23.782310 16835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:23.787968 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 16767
I20260926 14:39:23.788096 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:23.788708 16835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a: Bootstrap starting.
I20260926 14:39:23.789443 16747 external_mini_cluster.cc:1428] Setting key 6d8057443d605d01decfa608b0b9f1d7
I20260926 14:39:23.791111 16835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:23.791887 16835 log.cc:824] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:23.794046 16835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a: No bootstrap required, opened a new log
I20260926 14:39:23.798872 16835 raft_consensus.cc:348] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86c62766fc648b0a00db9f59723f86a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40107 } }
I20260926 14:39:23.799211 16835 raft_consensus.cc:374] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:23.799283 16835 raft_consensus.cc:729] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d86c62766fc648b0a00db9f59723f86a, State: Initialized, Role: FOLLOWER
I20260926 14:39:23.800051 16835 consensus_queue.cc:260] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86c62766fc648b0a00db9f59723f86a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40107 } }
I20260926 14:39:23.800218 16835 raft_consensus.cc:388] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:23.800284 16835 raft_consensus.cc:482] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:23.800376 16835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:23.801183 16835 raft_consensus.cc:504] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86c62766fc648b0a00db9f59723f86a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40107 } }
I20260926 14:39:23.801764 16835 leader_election.cc:302] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d86c62766fc648b0a00db9f59723f86a; no voters:
I20260926 14:39:23.802131 16835 leader_election.cc:288] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:23.802215 16840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:23.802438 16840 raft_consensus.cc:686] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [term 1 LEADER]: Becoming Leader. State: Replica: d86c62766fc648b0a00db9f59723f86a, State: Running, Role: LEADER
I20260926 14:39:23.802847 16840 consensus_queue.cc:237] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86c62766fc648b0a00db9f59723f86a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40107 } }
I20260926 14:39:23.803390 16835 sys_catalog.cc:564] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:23.805193 16841 sys_catalog.cc:455] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d86c62766fc648b0a00db9f59723f86a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86c62766fc648b0a00db9f59723f86a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40107 } } }
I20260926 14:39:23.805333 16841 sys_catalog.cc:458] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:23.806133 16842 sys_catalog.cc:455] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d86c62766fc648b0a00db9f59723f86a. Latest consensus state: current_term: 1 leader_uuid: "d86c62766fc648b0a00db9f59723f86a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86c62766fc648b0a00db9f59723f86a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40107 } } }
I20260926 14:39:23.806278 16842 sys_catalog.cc:458] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a [sys.catalog]: This master's current role is: LEADER
Sep 26 14:39:23 dist-test-slave-x4sn krb5kdc[16754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
W20260926 14:39:23.809785 16855 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:39:23.809922 16855 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:39:23.810019 16854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:23.810879 16854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:23.816349 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:23.792495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.16.90.254:40107)
0926 14:39:23.792913 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:23.792931 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:23.794183 (+ 1252us) server_negotiation.cc:407] Connection header received
0926 14:39:23.798394 (+ 4211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:23.798415 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:23.798802 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:23.799160 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:23.801391 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:23.802929 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:23.804878 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:23.805579 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:23.805738 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:23.810798 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:23.810805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:23.810821 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:23.810851 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:23.812925 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:23.813411 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:23.813415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:23.813419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:23.813522 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:23.813843 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:23.813847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:23.813848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:23.814329 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:23.814507 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:23.815058 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 14:39:23.815321 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260926 14:39:23.818040 16854 catalog_manager.cc:1347] Generated new cluster ID: d42267468aa145afaf2c462894def2a7
I20260926 14:39:23.818158 16854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:23.837752 16854 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:23.838877 16854 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:23.850471 16854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d86c62766fc648b0a00db9f59723f86a: Generated new TSK 0
I20260926 14:39:23.851492 16854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:39:23.880925 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40107
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:23.994328 16861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:23.994612 16861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:23.994676 16861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:23.998824 16861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:23.998894 16861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:23.998978 16861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:24.004482 16861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40107
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:24.005651 16861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:24.006691 16861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:24.016537 16866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:24.016527 16867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:24.016590 16869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:24.016716 16861 server_base.cc:1034] running on GCE node
I20260926 14:39:24.017175 16861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:24.017791 16861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:24.019042 16861 hybrid_clock.cc:648] HybridClock initialized: now 1790433564018960 us; error 30 us; skew 500 ppm
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[16754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:24.022050 16861 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:39:24.022887 16861 webserver.cc:460] Webserver started at http://127.16.90.193:44579/ using document root <none> and password file <none>
I20260926 14:39:24.023403 16861 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:24.023468 16861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:24.023648 16861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:24.025700 16861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "47afd6ef0f0c484885a45a9b10257076"
format_stamp: "Formatted at 2026-09-26 14:39:24 on dist-test-slave-x4sn"
server_key: "13b00a6220fbbd36803f9587e66ac81b"
server_key_iv: "b2e69343ed29c37415b0f08d598c352e"
server_key_version: "encryptionkey@0"
I20260926 14:39:24.026284 16861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "47afd6ef0f0c484885a45a9b10257076"
format_stamp: "Formatted at 2026-09-26 14:39:24 on dist-test-slave-x4sn"
server_key: "13b00a6220fbbd36803f9587e66ac81b"
server_key_iv: "b2e69343ed29c37415b0f08d598c352e"
server_key_version: "encryptionkey@0"
I20260926 14:39:24.029860 16861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:39:24.032226 16876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:24.033386 16861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:24.033488 16861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "47afd6ef0f0c484885a45a9b10257076"
format_stamp: "Formatted at 2026-09-26 14:39:24 on dist-test-slave-x4sn"
server_key: "13b00a6220fbbd36803f9587e66ac81b"
server_key_iv: "b2e69343ed29c37415b0f08d598c352e"
server_key_version: "encryptionkey@0"
I20260926 14:39:24.033574 16861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:24.054318 16861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:24.057282 16861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:24.057456 16861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:24.058117 16861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:24.059063 16861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:24.059120 16861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:24.059172 16861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:24.059201 16861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:24.068733 16861 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:37307
I20260926 14:39:24.068795 16989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:37307 every 8 connection(s)
I20260926 14:39:24.069607 16861 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:39:24.077497 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 16861
I20260926 14:39:24.077593 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:39:24.077824 16747 external_mini_cluster.cc:1428] Setting key 399a20480ad1971caa15bfadcc40e231
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[16754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:24.084044 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:24.071565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:33321 (local address 127.16.90.254:40107)
0926 14:39:24.071757 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:24.071760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:24.072564 (+ 804us) server_negotiation.cc:407] Connection header received
0926 14:39:24.073943 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:24.073947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:24.074010 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:24.074097 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:24.075302 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.075884 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.077048 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.077285 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.077407 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:24.079960 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:24.079963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:24.079965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:24.079991 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:24.081667 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:24.082337 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:24.082341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:24.082342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:24.082444 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:24.082794 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:24.082798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:24.082800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:24.083076 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:24.083196 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:24.083706 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 14:39:24.083854 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 14:39:24.084779 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:24.071958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40107 (local address 127.16.90.193:33321)
0926 14:39:24.072405 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:24.072449 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:24.073720 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:24.074281 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:24.074288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:24.074644 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:24.075132 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:24.075147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.076015 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.076017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:24.076905 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:24.076915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.077431 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.077436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:24.077623 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:24.078215 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:24.078233 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:24.079792 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:24.081841 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:24.081846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:24.081858 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:24.082229 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:24.082553 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:24.082556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:24.082557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:24.082704 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:24.083199 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:24.083204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:24.083447 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 14:39:24.083608 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 14:39:24.083926 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
I20260926 14:39:24.085969 16990 heartbeater.cc:347] Connected to a master server at 127.16.90.254:40107
I20260926 14:39:24.086221 16990 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:24.086678 16990 heartbeater.cc:511] Master 127.16.90.254:40107 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
I20260926 14:39:24.088634 16799 ts_manager.cc:194] Registered new tserver with Master: 47afd6ef0f0c484885a45a9b10257076 (127.16.90.193:37307)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
I20260926 14:39:24.090056 16799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:33321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:39:24.104101 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40107
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:24.218722 16995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:24.219012 16995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:24.219072 16995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:24.223021 16995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:24.223078 16995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:24.223162 16995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:24.228278 16995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40107
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:24.229331 16995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:24.230310 16995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:24.239573 17003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:24.239701 16995 server_base.cc:1034] running on GCE node
W20260926 14:39:24.239570 17000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:24.239573 17001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:24.240216 16995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:24.240784 16995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:24.241925 16995 hybrid_clock.cc:648] HybridClock initialized: now 1790433564241903 us; error 34 us; skew 500 ppm
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[16754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:24.244501 16995 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:39:24.245323 16995 webserver.cc:460] Webserver started at http://127.16.90.194:40933/ using document root <none> and password file <none>
I20260926 14:39:24.245817 16995 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:24.245872 16995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:24.246032 16995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:24.248075 16995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "07b86f9b13e449b78030b78851fe5277"
format_stamp: "Formatted at 2026-09-26 14:39:24 on dist-test-slave-x4sn"
server_key: "54405d45748357dda5a823dc96f415a8"
server_key_iv: "62dbd9e72483f0741765d2f646a92545"
server_key_version: "encryptionkey@0"
I20260926 14:39:24.248620 16995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "07b86f9b13e449b78030b78851fe5277"
format_stamp: "Formatted at 2026-09-26 14:39:24 on dist-test-slave-x4sn"
server_key: "54405d45748357dda5a823dc96f415a8"
server_key_iv: "62dbd9e72483f0741765d2f646a92545"
server_key_version: "encryptionkey@0"
I20260926 14:39:24.252254 16995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:24.254487 17010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:24.255703 16995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:24.255802 16995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "07b86f9b13e449b78030b78851fe5277"
format_stamp: "Formatted at 2026-09-26 14:39:24 on dist-test-slave-x4sn"
server_key: "54405d45748357dda5a823dc96f415a8"
server_key_iv: "62dbd9e72483f0741765d2f646a92545"
server_key_version: "encryptionkey@0"
I20260926 14:39:24.255896 16995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:24.269917 16995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:24.272557 16995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:24.272723 16995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:24.273343 16995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:24.274127 16995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:24.274179 16995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:24.274221 16995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:24.274250 16995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:24.282630 16995 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:42705
I20260926 14:39:24.282676 17123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:42705 every 8 connection(s)
I20260926 14:39:24.283463 16995 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:39:24.289871 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 16995
I20260926 14:39:24.289965 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:39:24.290195 16747 external_mini_cluster.cc:1428] Setting key 7e6a776f5ea97df78f8209f6bcde3f82
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[16754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:24.296804 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:24.285226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:51983 (local address 127.16.90.254:40107)
0926 14:39:24.285390 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:24.285395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:24.286203 (+ 808us) server_negotiation.cc:407] Connection header received
0926 14:39:24.287451 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:24.287454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:24.287500 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:24.287573 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:24.288676 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.289202 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.290319 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.290549 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.290649 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:24.293319 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:24.293323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:24.293326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:24.293353 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:24.294820 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:24.295276 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:24.295279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:24.295281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:24.295370 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:24.295694 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:24.295697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:24.295699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:24.295912 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:24.296011 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:24.296459 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 14:39:24.296600 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 14:39:24.297554 17126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:24.285613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40107 (local address 127.16.90.194:51983)
0926 14:39:24.286054 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:24.286094 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:24.287253 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:24.287739 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:24.287746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:24.288073 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:24.288554 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:24.288566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.289297 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.289299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:24.290196 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:24.290205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.290732 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.290737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:24.290921 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:24.291474 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:24.291494 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:24.293160 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:24.294934 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:24.294938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:24.294950 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:24.295188 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:24.295459 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:24.295462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:24.295464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:24.295591 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:24.296001 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:24.296006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:24.296237 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 14:39:24.296390 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 14:39:24.296711 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":126,"threads_started":1}
I20260926 14:39:24.298657 17124 heartbeater.cc:347] Connected to a master server at 127.16.90.254:40107
I20260926 14:39:24.298914 17124 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
I20260926 14:39:24.299453 17124 heartbeater.cc:511] Master 127.16.90.254:40107 requested a full tablet report, sending...
I20260926 14:39:24.300483 16799 ts_manager.cc:194] Registered new tserver with Master: 07b86f9b13e449b78030b78851fe5277 (127.16.90.194:42705)
I20260926 14:39:24.301128 16799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:51983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:39:24.314817 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40107
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:24.426901 17129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:24.427152 17129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:24.427218 17129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:24.431203 17129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:24.431257 17129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:24.431334 17129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:24.436406 17129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40107
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:24.437377 17129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:24.438393 17129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:24.447288 17135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:24.447310 17134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:24.447314 17137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:24.447386 17129 server_base.cc:1034] running on GCE node
I20260926 14:39:24.447885 17129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:24.448467 17129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:24.449631 17129 hybrid_clock.cc:648] HybridClock initialized: now 1790433564449619 us; error 30 us; skew 500 ppm
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[16754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:24.452239 17129 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:39:24.453018 17129 webserver.cc:460] Webserver started at http://127.16.90.195:35669/ using document root <none> and password file <none>
I20260926 14:39:24.453512 17129 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:24.453575 17129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:24.453747 17129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:24.455603 17129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "7177c31c5553435697199a71836a1d50"
format_stamp: "Formatted at 2026-09-26 14:39:24 on dist-test-slave-x4sn"
server_key: "e7a7460ce04fbda8209fe32fbe072e42"
server_key_iv: "a81f1b572e02729088857325dad88655"
server_key_version: "encryptionkey@0"
I20260926 14:39:24.456151 17129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "7177c31c5553435697199a71836a1d50"
format_stamp: "Formatted at 2026-09-26 14:39:24 on dist-test-slave-x4sn"
server_key: "e7a7460ce04fbda8209fe32fbe072e42"
server_key_iv: "a81f1b572e02729088857325dad88655"
server_key_version: "encryptionkey@0"
I20260926 14:39:24.459630 17129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:39:24.461884 17144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:24.463037 17129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:24.463145 17129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "7177c31c5553435697199a71836a1d50"
format_stamp: "Formatted at 2026-09-26 14:39:24 on dist-test-slave-x4sn"
server_key: "e7a7460ce04fbda8209fe32fbe072e42"
server_key_iv: "a81f1b572e02729088857325dad88655"
server_key_version: "encryptionkey@0"
I20260926 14:39:24.463234 17129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:24.478852 17129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:24.481755 17129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:24.481925 17129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:24.482491 17129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:24.483287 17129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:24.483335 17129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:24.483385 17129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:24.483412 17129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:24.491943 17129 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:46205
I20260926 14:39:24.491998 17257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:46205 every 8 connection(s)
I20260926 14:39:24.492789 17129 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:24.500519 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 17129
I20260926 14:39:24.500618 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:39:24.500854 16747 external_mini_cluster.cc:1428] Setting key cd8d6c26ca6597820ab5c905942d0468
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[16754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:24.505668 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:24.494480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:48277 (local address 127.16.90.254:40107)
0926 14:39:24.494624 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:24.494628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:24.495525 (+ 897us) server_negotiation.cc:407] Connection header received
0926 14:39:24.496838 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:24.496841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:24.496887 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:24.496960 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:24.498043 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.498555 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.499632 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.499901 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.499980 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:24.502463 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:24.502467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:24.502470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:24.502493 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:24.503933 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:24.504386 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:24.504389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:24.504391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:24.504470 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:24.504741 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:24.504743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:24.504744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:24.504942 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:24.505018 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:24.505459 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 14:39:24.505556 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:39:24.506465 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:24.494884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40107 (local address 127.16.90.195:48277)
0926 14:39:24.495374 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:24.495417 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:24.496630 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:24.497120 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:24.497126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:24.497471 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:24.497933 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:24.497947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.498667 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.498670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:24.499532 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:24.499541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.500014 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.500017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:24.500176 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:24.500720 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:24.500738 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:24.502332 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:24.504022 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:24.504027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:24.504038 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:24.504291 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:24.504531 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:24.504534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:24.504535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:24.504670 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:24.505026 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:24.505030 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:24.505252 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 14:39:24.505408 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 14:39:24.505672 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":155,"threads_started":1}
I20260926 14:39:24.507441 17258 heartbeater.cc:347] Connected to a master server at 127.16.90.254:40107
I20260926 14:39:24.507658 17258 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:24.508121 17258 heartbeater.cc:511] Master 127.16.90.254:40107 requested a full tablet report, sending...
I20260926 14:39:24.509016 16799 ts_manager.cc:194] Registered new tserver with Master: 7177c31c5553435697199a71836a1d50 (127.16.90.195:46205)
I20260926 14:39:24.509536 16799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:48277
I20260926 14:39:24.514778 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[16754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433564, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[16754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433564, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:39:24.539067 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:24.530280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.16.90.193:37307)
0926 14:39:24.530623 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:24.530635 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:24.530661 (+ 26us) server_negotiation.cc:407] Connection header received
0926 14:39:24.530719 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:24.530726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:24.530906 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:24.531033 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:24.531660 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.532467 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.533321 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.533675 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.533768 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:24.536013 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:24.536021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:24.536032 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:24.536057 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:24.537464 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:24.537869 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:24.537876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:24.537878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:24.537985 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:24.538236 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:24.538238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:24.538239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:24.538471 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:24.538629 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:24.538761 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 14:39:24.538894 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":136,"threads_started":1}
W20260926 14:39:24.540102 16904 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44966
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[16754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[16754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:39:24.562837 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:24.555632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.16.90.193:37307)
0926 14:39:24.555773 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:24.555777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:24.555897 (+ 120us) server_negotiation.cc:407] Connection header received
0926 14:39:24.555993 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:24.555996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:24.556035 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:24.556100 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:24.556688 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.557173 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.557915 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:24.558120 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:24.558198 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:24.560382 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:24.560385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:24.560387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:24.560407 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:24.561451 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:24.561819 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:24.561822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:24.561823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:24.561891 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:24.562115 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:24.562119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:24.562120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:24.562269 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:24.562374 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:24.562591 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 14:39:24.562695 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 14:39:24.563871 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 16861
I20260926 14:39:24.568918 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 16995
I20260926 14:39:24.573588 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 17129
I20260926 14:39:24.578162 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 16767
2026-09-26T14:39:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2151 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[17272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[17272](info): set up 2 sockets
Sep 26 14:39:24 dist-test-slave-x4sn krb5kdc[17272](info): commencing operation
krb5kdc: starting...
W20260926 14:39:25.532724 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:39:25 dist-test-slave-x4sn krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:39:25.648526 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:41613
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:41613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:25.756332 17285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:25.756618 17285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:25.756673 17285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:25.760664 17285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:25.760730 17285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:25.760757 17285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:25.760776 17285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:25.760798 17285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:25.766095 17285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:41613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:41613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:25.767176 17285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:25.768218 17285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:25.776719 17290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:25.776719 17293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:25.776779 17291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:25.776794 17285 server_base.cc:1034] running on GCE node
I20260926 14:39:25.777573 17285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:25.778663 17285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:25.779810 17285 hybrid_clock.cc:648] HybridClock initialized: now 1790433565779790 us; error 33 us; skew 500 ppm
Sep 26 14:39:25 dist-test-slave-x4sn krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433565, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:25.782395 17285 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:39:25.783280 17285 webserver.cc:460] Webserver started at http://127.16.90.254:44713/ using document root <none> and password file <none>
I20260926 14:39:25.783870 17285 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:25.783938 17285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:25.784142 17285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:25.786288 17285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "3e184a8de3a04122b2c85dc03784a5ec"
format_stamp: "Formatted at 2026-09-26 14:39:25 on dist-test-slave-x4sn"
server_key: "6396f07c297bc92e732cf797323a96ee"
server_key_iv: "dff6086713e9039542e4340ddf712417"
server_key_version: "encryptionkey@0"
I20260926 14:39:25.786846 17285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "3e184a8de3a04122b2c85dc03784a5ec"
format_stamp: "Formatted at 2026-09-26 14:39:25 on dist-test-slave-x4sn"
server_key: "6396f07c297bc92e732cf797323a96ee"
server_key_iv: "dff6086713e9039542e4340ddf712417"
server_key_version: "encryptionkey@0"
I20260926 14:39:25.790683 17285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:39:25.793161 17300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:25.794431 17285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:39:25.794534 17285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "3e184a8de3a04122b2c85dc03784a5ec"
format_stamp: "Formatted at 2026-09-26 14:39:25 on dist-test-slave-x4sn"
server_key: "6396f07c297bc92e732cf797323a96ee"
server_key_iv: "dff6086713e9039542e4340ddf712417"
server_key_version: "encryptionkey@0"
I20260926 14:39:25.794627 17285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:25.809826 17285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:25.812883 17285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:25.813042 17285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:25.820024 17285 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:41613
I20260926 14:39:25.820108 17352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:41613 every 8 connection(s)
I20260926 14:39:25.821007 17285 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:25.823910 17353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:25.823906 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 17285
I20260926 14:39:25.824002 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:25.824245 16747 external_mini_cluster.cc:1428] Setting key 49bcda560351e3045906ddbd1810bcc4
I20260926 14:39:25.831019 17353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec: Bootstrap starting.
Sep 26 14:39:25 dist-test-slave-x4sn krb5kdc[17272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:25.833817 17353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:25.834641 17353 log.cc:824] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:25.836874 17353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec: No bootstrap required, opened a new log
I20260926 14:39:25.838930 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:25.825654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44920 (local address 127.16.90.254:41613)
0926 14:39:25.826091 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:25.826109 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:25.826152 (+ 43us) server_negotiation.cc:407] Connection header received
0926 14:39:25.827107 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:25.827132 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:25.827579 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:25.827981 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:25.828601 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:25.830098 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:25.830912 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:25.831569 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:25.831745 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:25.834167 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:25.834173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:25.834188 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:25.834218 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:25.836038 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:25.836447 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:25.836452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:25.836458 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:25.836605 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:25.836827 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:25.836829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:25.836831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:25.837315 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:25.837505 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:25.837693 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 14:39:25.837979 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260926 14:39:25.841261 17353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e184a8de3a04122b2c85dc03784a5ec" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41613 } }
I20260926 14:39:25.841523 17353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:25.841581 17353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e184a8de3a04122b2c85dc03784a5ec, State: Initialized, Role: FOLLOWER
I20260926 14:39:25.841991 17353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e184a8de3a04122b2c85dc03784a5ec" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41613 } }
I20260926 14:39:25.842127 17353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:25.842190 17353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:25.842281 17353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:25.843137 17353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e184a8de3a04122b2c85dc03784a5ec" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41613 } }
I20260926 14:39:25.843442 17353 leader_election.cc:302] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e184a8de3a04122b2c85dc03784a5ec; no voters:
I20260926 14:39:25.843742 17353 leader_election.cc:288] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:25.843880 17358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:25.844095 17358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [term 1 LEADER]: Becoming Leader. State: Replica: 3e184a8de3a04122b2c85dc03784a5ec, State: Running, Role: LEADER
I20260926 14:39:25.844401 17358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e184a8de3a04122b2c85dc03784a5ec" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41613 } }
I20260926 14:39:25.844981 17353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:25.845445 17360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e184a8de3a04122b2c85dc03784a5ec. Latest consensus state: current_term: 1 leader_uuid: "3e184a8de3a04122b2c85dc03784a5ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e184a8de3a04122b2c85dc03784a5ec" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41613 } } }
I20260926 14:39:25.845549 17360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:25.845880 17359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e184a8de3a04122b2c85dc03784a5ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e184a8de3a04122b2c85dc03784a5ec" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41613 } } }
I20260926 14:39:25.846004 17359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:25.845942 17363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:25.849797 17363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:25.855266 17363 catalog_manager.cc:1347] Generated new cluster ID: 91d3117247bb423a86366f157237cc3a
I20260926 14:39:25.855338 17363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:25.865094 17363 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:25.866310 17363 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:25.873478 17363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e184a8de3a04122b2c85dc03784a5ec: Generated new TSK 0
I20260926 14:39:25.874141 17363 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:39:25.905921 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:41613
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:26.012418 17379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:26.012676 17379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:26.012720 17379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:26.016837 17379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:26.016906 17379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:26.016966 17379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:26.022302 17379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:41613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:26.023403 17379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:26.024469 17379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:26.033583 17384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:26.033577 17385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:26.033622 17379 server_base.cc:1034] running on GCE node
W20260926 14:39:26.033577 17387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:26.034119 17379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:26.034730 17379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:26.035883 17379 hybrid_clock.cc:648] HybridClock initialized: now 1790433566035867 us; error 29 us; skew 500 ppm
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:26.038415 17379 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:39:26.039213 17379 webserver.cc:460] Webserver started at http://127.16.90.193:38651/ using document root <none> and password file <none>
I20260926 14:39:26.039765 17379 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:26.039839 17379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:26.040032 17379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:26.042019 17379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "96e599d7ad594f7c930935d23381034e"
format_stamp: "Formatted at 2026-09-26 14:39:26 on dist-test-slave-x4sn"
server_key: "d8040a4d776a63b8fb2e5096fc535041"
server_key_iv: "9e90c5170a5a134dae5fedc4d8404b0b"
server_key_version: "encryptionkey@0"
I20260926 14:39:26.042537 17379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "96e599d7ad594f7c930935d23381034e"
format_stamp: "Formatted at 2026-09-26 14:39:26 on dist-test-slave-x4sn"
server_key: "d8040a4d776a63b8fb2e5096fc535041"
server_key_iv: "9e90c5170a5a134dae5fedc4d8404b0b"
server_key_version: "encryptionkey@0"
I20260926 14:39:26.046074 17379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:39:26.048360 17394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:26.049523 17379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:26.049624 17379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "96e599d7ad594f7c930935d23381034e"
format_stamp: "Formatted at 2026-09-26 14:39:26 on dist-test-slave-x4sn"
server_key: "d8040a4d776a63b8fb2e5096fc535041"
server_key_iv: "9e90c5170a5a134dae5fedc4d8404b0b"
server_key_version: "encryptionkey@0"
I20260926 14:39:26.049712 17379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:26.066876 17379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:26.069458 17379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:26.069620 17379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:26.070163 17379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:26.070942 17379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:26.070989 17379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:26.071038 17379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:26.071065 17379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:26.079140 17379 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:34885
I20260926 14:39:26.079167 17507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:34885 every 8 connection(s)
I20260926 14:39:26.079970 17379 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:39:26.081459 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 17379
I20260926 14:39:26.081568 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:39:26.081837 16747 external_mini_cluster.cc:1428] Setting key f22e20675d404992d1047abcd6797a6b
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
I20260926 14:39:26.093622 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.081699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:37419 (local address 127.16.90.254:41613)
0926 14:39:26.081858 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:26.081863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:26.082689 (+ 826us) server_negotiation.cc:407] Connection header received
0926 14:39:26.083742 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:26.083745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:26.083798 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:26.083871 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:26.085228 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.085751 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.087298 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.087537 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.087621 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.090138 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:26.090142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:26.090143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:26.090174 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:26.091586 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.092069 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.092072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.092074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.092152 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.092519 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.092522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.092524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.092740 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:26.092833 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:26.093333 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 14:39:26.093459 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 14:39:26.094376 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.082071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:41613 (local address 127.16.90.193:37419)
0926 14:39:26.082531 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:26.082588 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:26.083544 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:26.084043 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:26.084049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:26.084435 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:26.085074 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.085092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.085875 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.085878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.087172 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.087194 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.087711 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.087717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.087911 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.088475 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:26.088495 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:26.089982 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:26.091723 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:26.091728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:26.091742 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:26.091973 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:26.092262 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:26.092265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:26.092268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:26.092422 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:26.092835 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:26.092840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:26.093078 (+ 238us) client_negotiation.cc:769] Sending connection context
0926 14:39:26.093233 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 14:39:26.093466 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
I20260926 14:39:26.095574 17508 heartbeater.cc:347] Connected to a master server at 127.16.90.254:41613
I20260926 14:39:26.095814 17508 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:26.096268 17508 heartbeater.cc:511] Master 127.16.90.254:41613 requested a full tablet report, sending...
I20260926 14:39:26.097677 17317 ts_manager.cc:194] Registered new tserver with Master: 96e599d7ad594f7c930935d23381034e (127.16.90.193:34885)
I20260926 14:39:26.098816 17317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:37419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:39:26.106968 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:41613
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:26.215634 17513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:26.215920 17513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:26.215961 17513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:26.219833 17513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:26.219880 17513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:26.219936 17513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:26.225078 17513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:41613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:26.226037 17513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:26.226976 17513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:26.236083 17519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:26.236100 17521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:26.236109 17518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:26.236385 17513 server_base.cc:1034] running on GCE node
I20260926 14:39:26.236783 17513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:26.237377 17513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:26.238528 17513 hybrid_clock.cc:648] HybridClock initialized: now 1790433566238515 us; error 26 us; skew 500 ppm
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:26.241204 17513 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:39:26.242050 17513 webserver.cc:460] Webserver started at http://127.16.90.194:44489/ using document root <none> and password file <none>
I20260926 14:39:26.242556 17513 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:26.242619 17513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:26.242794 17513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:26.244817 17513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "93b0d69418ee4b1ba7091e9474db72da"
format_stamp: "Formatted at 2026-09-26 14:39:26 on dist-test-slave-x4sn"
server_key: "3027cd6ddf8aa2d08245e6b3f45931fc"
server_key_iv: "9044740781d2616f08108df8f82d9581"
server_key_version: "encryptionkey@0"
I20260926 14:39:26.245344 17513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "93b0d69418ee4b1ba7091e9474db72da"
format_stamp: "Formatted at 2026-09-26 14:39:26 on dist-test-slave-x4sn"
server_key: "3027cd6ddf8aa2d08245e6b3f45931fc"
server_key_iv: "9044740781d2616f08108df8f82d9581"
server_key_version: "encryptionkey@0"
I20260926 14:39:26.248845 17513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:26.251055 17528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:26.252295 17513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:26.252393 17513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "93b0d69418ee4b1ba7091e9474db72da"
format_stamp: "Formatted at 2026-09-26 14:39:26 on dist-test-slave-x4sn"
server_key: "3027cd6ddf8aa2d08245e6b3f45931fc"
server_key_iv: "9044740781d2616f08108df8f82d9581"
server_key_version: "encryptionkey@0"
I20260926 14:39:26.252477 17513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:26.262426 17513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:26.264956 17513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:26.265115 17513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:26.265640 17513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:26.266376 17513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:26.266422 17513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:26.266469 17513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:26.266497 17513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:26.275193 17513 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:41557
I20260926 14:39:26.275224 17641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:41557 every 8 connection(s)
I20260926 14:39:26.276072 17513 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:39:26.282420 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 17513
I20260926 14:39:26.282531 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:39:26.282775 16747 external_mini_cluster.cc:1428] Setting key 1a0de747f5a088faa86fcc99de731bd6
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:26.289312 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.277801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:40431 (local address 127.16.90.254:41613)
0926 14:39:26.277969 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:26.277975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:26.278772 (+ 797us) server_negotiation.cc:407] Connection header received
0926 14:39:26.279761 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:26.279763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:26.279811 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:26.279884 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:26.281119 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.281660 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.282836 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.283133 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.283221 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.285848 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:26.285852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:26.285855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:26.285882 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:26.287292 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.287843 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.287846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.287847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.287939 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.288275 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.288279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.288280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.288459 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:26.288541 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:26.289022 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 14:39:26.289154 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 14:39:26.290126 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.278173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:41613 (local address 127.16.90.194:40431)
0926 14:39:26.278621 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:26.278663 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:26.279560 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:26.280105 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:26.280113 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:26.280499 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:26.280971 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.280985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.281784 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.281786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.282722 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.282731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.283292 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.283297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.283460 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.284075 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:26.284096 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:26.285685 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:26.287411 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:26.287415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:26.287431 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:26.287744 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:26.288028 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:26.288031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:26.288033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:26.288179 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:26.288553 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:26.288560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:26.288795 (+ 235us) client_negotiation.cc:769] Sending connection context
0926 14:39:26.288950 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 14:39:26.289302 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":103,"threads_started":1}
I20260926 14:39:26.291339 17642 heartbeater.cc:347] Connected to a master server at 127.16.90.254:41613
I20260926 14:39:26.291553 17642 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
I20260926 14:39:26.292058 17642 heartbeater.cc:511] Master 127.16.90.254:41613 requested a full tablet report, sending...
I20260926 14:39:26.293077 17317 ts_manager.cc:194] Registered new tserver with Master: 93b0d69418ee4b1ba7091e9474db72da (127.16.90.194:41557)
I20260926 14:39:26.293625 17317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:40431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:39:26.308506 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:41613
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:26.419215 17647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:26.419464 17647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:26.419505 17647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:26.423461 17647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:26.423516 17647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:26.423571 17647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:26.428603 17647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:41613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:26.429563 17647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:26.430545 17647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:26.439540 17653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:26.439538 17652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:26.439517 17655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:26.439719 17647 server_base.cc:1034] running on GCE node
I20260926 14:39:26.440166 17647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:26.440760 17647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:26.441919 17647 hybrid_clock.cc:648] HybridClock initialized: now 1790433566441900 us; error 29 us; skew 500 ppm
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:26.444520 17647 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:39:26.445399 17647 webserver.cc:460] Webserver started at http://127.16.90.195:42765/ using document root <none> and password file <none>
I20260926 14:39:26.445932 17647 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:26.445990 17647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:26.446229 17647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:26.448208 17647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "b9c2816961c34c49b60796043afe7deb"
format_stamp: "Formatted at 2026-09-26 14:39:26 on dist-test-slave-x4sn"
server_key: "3fe45165c2846a26e831d8b39d0c663e"
server_key_iv: "bce38a1ff204dcf915879f2b81795401"
server_key_version: "encryptionkey@0"
I20260926 14:39:26.448696 17647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "b9c2816961c34c49b60796043afe7deb"
format_stamp: "Formatted at 2026-09-26 14:39:26 on dist-test-slave-x4sn"
server_key: "3fe45165c2846a26e831d8b39d0c663e"
server_key_iv: "bce38a1ff204dcf915879f2b81795401"
server_key_version: "encryptionkey@0"
I20260926 14:39:26.452147 17647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:26.454452 17662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:26.455600 17647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:26.455730 17647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "b9c2816961c34c49b60796043afe7deb"
format_stamp: "Formatted at 2026-09-26 14:39:26 on dist-test-slave-x4sn"
server_key: "3fe45165c2846a26e831d8b39d0c663e"
server_key_iv: "bce38a1ff204dcf915879f2b81795401"
server_key_version: "encryptionkey@0"
I20260926 14:39:26.455821 17647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:26.472429 17647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:26.475224 17647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:26.475368 17647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:26.475917 17647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:26.476660 17647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:26.476703 17647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:26.476740 17647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:26.476754 17647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:26.485360 17647 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:33989
I20260926 14:39:26.485391 17775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:33989 every 8 connection(s)
I20260926 14:39:26.486212 17647 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:26.494050 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 17647
I20260926 14:39:26.494150 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:39:26.494376 16747 external_mini_cluster.cc:1428] Setting key 15ce7b4fe8ae400cc21bf299b7264c14
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433566, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:26.498798 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.488018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:46017 (local address 127.16.90.254:41613)
0926 14:39:26.488181 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:26.488185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:26.488965 (+ 780us) server_negotiation.cc:407] Connection header received
0926 14:39:26.489807 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:26.489809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:26.489856 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:26.489927 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:26.491041 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.491556 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.492639 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.492847 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.492923 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.495280 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:26.495284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:26.495287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:26.495315 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:26.496765 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.497290 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.497292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.497294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.497375 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.497689 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.497691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.497692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.497877 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:26.497948 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:26.498479 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 14:39:26.498602 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:39:26.499596 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.488380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:41613 (local address 127.16.90.195:46017)
0926 14:39:26.488809 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:26.488849 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:26.489611 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:26.490084 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:26.490090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:26.490451 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:26.490922 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.490935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.491648 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.491650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.492542 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.492551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.492958 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.492961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.493096 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.493666 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:26.493683 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:26.495137 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:26.496886 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:26.496891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:26.496904 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:26.497170 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:26.497446 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:26.497448 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:26.497449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:26.497621 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:26.497957 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:26.497961 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:26.498244 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 14:39:26.498402 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 14:39:26.498676 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260926 14:39:26.500770 17776 heartbeater.cc:347] Connected to a master server at 127.16.90.254:41613
I20260926 14:39:26.501039 17776 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:26.501557 17776 heartbeater.cc:511] Master 127.16.90.254:41613 requested a full tablet report, sending...
I20260926 14:39:26.502650 17317 ts_manager.cc:194] Registered new tserver with Master: b9c2816961c34c49b60796043afe7deb (127.16.90.195:33989)
I20260926 14:39:26.503228 17317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:46017
I20260926 14:39:26.508282 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:39:26.518939 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.511895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.16.90.254:41613)
0926 14:39:26.512040 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:26.512045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:26.512181 (+ 136us) server_negotiation.cc:407] Connection header received
0926 14:39:26.512334 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:26.512337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:26.512381 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:26.512455 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:26.513085 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.513611 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.514419 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.514637 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.514711 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.516101 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:26.516104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:26.516106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:26.516129 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:26.517450 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.517854 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.517856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.517857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.517923 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.518219 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.518221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.518222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.518431 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:26.518530 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:26.518686 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 14:39:26.518780 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:39:26.524199 17317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:39:26.527168 17317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:26.542307 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.536800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39302 (local address 127.16.90.193:34885)
0926 14:39:26.537137 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:26.537150 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:26.537184 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:39:26.537252 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:26.537256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:26.537466 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:26.537615 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:26.538128 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.539252 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.540928 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.541623 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.541740 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.541895 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:26.542065 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 14:39:26.542162 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":122,"threads_started":1}
I20260926 14:39:26.542297 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.536563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:34885 (local address 127.0.0.1:39302)
0926 14:39:26.536913 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:26.536956 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:26.537106 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:26.537818 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:26.537824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:26.537849 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:26.538000 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.538008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.539382 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.539386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.540809 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.540820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.541757 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.541761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.541877 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.541914 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 14:39:26.542015 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 14:39:26.542138 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":48,"thread_start_us":89,"threads_started":1}
I20260926 14:39:26.542677 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.537573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:41557 (local address 127.0.0.1:41432)
0926 14:39:26.537981 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:26.537992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:26.538064 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:26.538627 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:26.538630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:26.538645 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:26.538750 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.538757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.539937 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.539940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.541118 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.541129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.542156 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.542160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.542351 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.542416 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 14:39:26.542473 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:39:26.542534 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":90,"thread_start_us":81,"threads_started":1}
I20260926 14:39:26.543243 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.537949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41432 (local address 127.16.90.194:41557)
0926 14:39:26.538275 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:26.538288 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:26.538306 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:39:26.538362 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:26.538365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:26.538504 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:26.538626 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:26.538872 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.539802 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.541231 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.542019 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.542835 (+ 816us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.542978 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:26.543022 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 14:39:26.543103 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
I20260926 14:39:26.543709 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.537277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33989 (local address 127.0.0.1:38824)
0926 14:39:26.537718 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:26.537730 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:26.537815 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:26.538649 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:26.538652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:26.538692 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:26.538870 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.538877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.539974 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.539978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.541089 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.541096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.542156 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.542160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.542356 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.543399 (+ 1043us) client_negotiation.cc:769] Sending connection context
0926 14:39:26.543467 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:39:26.543536 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":1157,"spinlock_wait_cycles":896,"thread_start_us":84,"threads_started":1}
I20260926 14:39:26.544340 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.537532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.16.90.195:33989)
0926 14:39:26.538218 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:26.538230 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:26.538250 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:39:26.538317 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:26.538322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:26.538500 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:26.538764 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:26.538970 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.539861 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.541189 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.542019 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.543920 (+ 1901us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.544065 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:26.544114 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 14:39:26.544202 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":149,"threads_started":1}
I20260926 14:39:26.545907 17576 tablet_service.cc:1467] Processing CreateTablet for tablet f7437f733d4e43a2b6e6f1b999dca820 (DEFAULT_TABLE table=test-table [id=c4b22e1e20534ae7ad3c397725921447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:26.545980 17442 tablet_service.cc:1467] Processing CreateTablet for tablet f7437f733d4e43a2b6e6f1b999dca820 (DEFAULT_TABLE table=test-table [id=c4b22e1e20534ae7ad3c397725921447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:26.546893 17576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7437f733d4e43a2b6e6f1b999dca820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:26.546993 17442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7437f733d4e43a2b6e6f1b999dca820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:26.546950 17710 tablet_service.cc:1467] Processing CreateTablet for tablet f7437f733d4e43a2b6e6f1b999dca820 (DEFAULT_TABLE table=test-table [id=c4b22e1e20534ae7ad3c397725921447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:26.548305 17710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7437f733d4e43a2b6e6f1b999dca820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:26.553331 17793 tablet_bootstrap.cc:492] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da: Bootstrap starting.
I20260926 14:39:26.554672 17794 tablet_bootstrap.cc:492] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e: Bootstrap starting.
I20260926 14:39:26.554986 17795 tablet_bootstrap.cc:492] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb: Bootstrap starting.
I20260926 14:39:26.555976 17793 tablet_bootstrap.cc:654] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:26.556761 17794 tablet_bootstrap.cc:654] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:26.557073 17795 tablet_bootstrap.cc:654] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:26.557252 17793 log.cc:824] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:26.557775 17794 log.cc:824] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:26.558175 17795 log.cc:824] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:26.559638 17794 tablet_bootstrap.cc:492] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e: No bootstrap required, opened a new log
I20260926 14:39:26.559783 17793 tablet_bootstrap.cc:492] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da: No bootstrap required, opened a new log
I20260926 14:39:26.559864 17794 ts_tablet_manager.cc:1397] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260926 14:39:26.559926 17795 tablet_bootstrap.cc:492] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb: No bootstrap required, opened a new log
I20260926 14:39:26.559952 17793 ts_tablet_manager.cc:1397] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 14:39:26.560094 17795 ts_tablet_manager.cc:1397] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:39:26.562819 17795 raft_consensus.cc:348] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e599d7ad594f7c930935d23381034e" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34885 } } peers { permanent_uuid: "b9c2816961c34c49b60796043afe7deb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33989 } } peers { permanent_uuid: "93b0d69418ee4b1ba7091e9474db72da" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41557 } }
I20260926 14:39:26.563042 17795 raft_consensus.cc:374] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:26.563102 17795 raft_consensus.cc:729] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c2816961c34c49b60796043afe7deb, State: Initialized, Role: FOLLOWER
I20260926 14:39:26.563517 17795 consensus_queue.cc:260] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e599d7ad594f7c930935d23381034e" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34885 } } peers { permanent_uuid: "b9c2816961c34c49b60796043afe7deb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33989 } } peers { permanent_uuid: "93b0d69418ee4b1ba7091e9474db72da" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41557 } }
I20260926 14:39:26.563771 17794 raft_consensus.cc:348] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e599d7ad594f7c930935d23381034e" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34885 } } peers { permanent_uuid: "b9c2816961c34c49b60796043afe7deb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33989 } } peers { permanent_uuid: "93b0d69418ee4b1ba7091e9474db72da" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41557 } }
I20260926 14:39:26.563771 17793 raft_consensus.cc:348] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e599d7ad594f7c930935d23381034e" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34885 } } peers { permanent_uuid: "b9c2816961c34c49b60796043afe7deb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33989 } } peers { permanent_uuid: "93b0d69418ee4b1ba7091e9474db72da" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41557 } }
I20260926 14:39:26.564028 17794 raft_consensus.cc:374] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:26.564028 17793 raft_consensus.cc:374] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:26.564081 17794 raft_consensus.cc:729] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e599d7ad594f7c930935d23381034e, State: Initialized, Role: FOLLOWER
I20260926 14:39:26.564088 17793 raft_consensus.cc:729] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b0d69418ee4b1ba7091e9474db72da, State: Initialized, Role: FOLLOWER
I20260926 14:39:26.564358 17776 heartbeater.cc:503] Master 127.16.90.254:41613 was elected leader, sending a full tablet report...
I20260926 14:39:26.564527 17795 ts_tablet_manager.cc:1428] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 14:39:26.564535 17794 consensus_queue.cc:260] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e599d7ad594f7c930935d23381034e" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34885 } } peers { permanent_uuid: "b9c2816961c34c49b60796043afe7deb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33989 } } peers { permanent_uuid: "93b0d69418ee4b1ba7091e9474db72da" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41557 } }
I20260926 14:39:26.564535 17793 consensus_queue.cc:260] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e599d7ad594f7c930935d23381034e" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34885 } } peers { permanent_uuid: "b9c2816961c34c49b60796043afe7deb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33989 } } peers { permanent_uuid: "93b0d69418ee4b1ba7091e9474db72da" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41557 } }
I20260926 14:39:26.565446 17793 ts_tablet_manager.cc:1428] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:39:26.565727 17794 ts_tablet_manager.cc:1428] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260926 14:39:26.565757 17642 heartbeater.cc:503] Master 127.16.90.254:41613 was elected leader, sending a full tablet report...
I20260926 14:39:26.566035 17508 heartbeater.cc:503] Master 127.16.90.254:41613 was elected leader, sending a full tablet report...
W20260926 14:39:26.581079 17509 tablet.cc:2369] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:39:26.584582 17801 raft_consensus.cc:482] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:26.584797 17801 raft_consensus.cc:504] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e599d7ad594f7c930935d23381034e" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34885 } } peers { permanent_uuid: "b9c2816961c34c49b60796043afe7deb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33989 } } peers { permanent_uuid: "93b0d69418ee4b1ba7091e9474db72da" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41557 } }
I20260926 14:39:26.585875 17801 leader_election.cc:288] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9c2816961c34c49b60796043afe7deb (127.16.90.195:33989), 93b0d69418ee4b1ba7091e9474db72da (127.16.90.194:41557)
I20260926 14:39:26.589304 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.586164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:45395 (local address 127.16.90.195:33989)
0926 14:39:26.586283 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:26.586288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:26.586319 (+ 31us) server_negotiation.cc:407] Connection header received
0926 14:39:26.586457 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:26.586460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:26.586501 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:26.586559 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:26.586804 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.587300 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.588372 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.588792 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.588896 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.588950 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:26.589114 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 14:39:26.589184 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:39:26.589305 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.586099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33989 (local address 127.16.90.193:45395)
0926 14:39:26.586247 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:26.586264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:26.586357 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:26.586577 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:26.586580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:26.586597 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:26.586708 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.586713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.587400 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.587404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.588251 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.588261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.588896 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.588900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.589009 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.589028 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:39:26.589092 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:39:26.589174 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 14:39:26.589745 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.586394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:41557 (local address 127.16.90.193:58511)
0926 14:39:26.586760 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:26.586771 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:26.586860 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:26.587069 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:26.587072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:26.587087 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:26.587211 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.587216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.587917 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.587920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.588754 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:26.588763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.589370 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.589374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:26.589464 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.589485 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:39:26.589550 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:39:26.589612 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":122,"threads_started":1}
I20260926 14:39:26.589910 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.586533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:58511 (local address 127.16.90.194:41557)
0926 14:39:26.586646 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:26.586651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:26.586840 (+ 189us) server_negotiation.cc:407] Connection header received
0926 14:39:26.586940 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:26.586943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:26.586988 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:26.587050 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:26.587310 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.587815 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.588866 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.589256 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.589359 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.589404 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:26.589729 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 14:39:26.589807 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:39:26.590097 17730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7437f733d4e43a2b6e6f1b999dca820" candidate_uuid: "96e599d7ad594f7c930935d23381034e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c2816961c34c49b60796043afe7deb" is_pre_election: true
I20260926 14:39:26.590441 17730 raft_consensus.cc:2454] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96e599d7ad594f7c930935d23381034e in term 0.
I20260926 14:39:26.590462 17596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7437f733d4e43a2b6e6f1b999dca820" candidate_uuid: "96e599d7ad594f7c930935d23381034e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b0d69418ee4b1ba7091e9474db72da" is_pre_election: true
I20260926 14:39:26.590718 17596 raft_consensus.cc:2454] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96e599d7ad594f7c930935d23381034e in term 0.
I20260926 14:39:26.591009 17397 leader_election.cc:302] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96e599d7ad594f7c930935d23381034e, b9c2816961c34c49b60796043afe7deb; no voters:
I20260926 14:39:26.591354 17801 raft_consensus.cc:2781] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:26.591430 17801 raft_consensus.cc:482] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:26.591459 17801 raft_consensus.cc:3037] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:26.592550 17801 raft_consensus.cc:504] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e599d7ad594f7c930935d23381034e" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34885 } } peers { permanent_uuid: "b9c2816961c34c49b60796043afe7deb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33989 } } peers { permanent_uuid: "93b0d69418ee4b1ba7091e9474db72da" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41557 } }
I20260926 14:39:26.592880 17801 leader_election.cc:288] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [CANDIDATE]: Term 1 election: Requested vote from peers b9c2816961c34c49b60796043afe7deb (127.16.90.195:33989), 93b0d69418ee4b1ba7091e9474db72da (127.16.90.194:41557)
I20260926 14:39:26.593235 17730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7437f733d4e43a2b6e6f1b999dca820" candidate_uuid: "96e599d7ad594f7c930935d23381034e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c2816961c34c49b60796043afe7deb"
I20260926 14:39:26.593339 17596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7437f733d4e43a2b6e6f1b999dca820" candidate_uuid: "96e599d7ad594f7c930935d23381034e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b0d69418ee4b1ba7091e9474db72da"
I20260926 14:39:26.593362 17730 raft_consensus.cc:3037] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:26.593484 17596 raft_consensus.cc:3037] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:26.594465 17596 raft_consensus.cc:2454] T f7437f733d4e43a2b6e6f1b999dca820 P 93b0d69418ee4b1ba7091e9474db72da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96e599d7ad594f7c930935d23381034e in term 1.
I20260926 14:39:26.594470 17730 raft_consensus.cc:2454] T f7437f733d4e43a2b6e6f1b999dca820 P b9c2816961c34c49b60796043afe7deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96e599d7ad594f7c930935d23381034e in term 1.
I20260926 14:39:26.594733 17397 leader_election.cc:302] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93b0d69418ee4b1ba7091e9474db72da, 96e599d7ad594f7c930935d23381034e; no voters:
I20260926 14:39:26.594897 17801 raft_consensus.cc:2781] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:26.595109 17801 raft_consensus.cc:686] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [term 1 LEADER]: Becoming Leader. State: Replica: 96e599d7ad594f7c930935d23381034e, State: Running, Role: LEADER
I20260926 14:39:26.595516 17801 consensus_queue.cc:237] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e599d7ad594f7c930935d23381034e" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34885 } } peers { permanent_uuid: "b9c2816961c34c49b60796043afe7deb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33989 } } peers { permanent_uuid: "93b0d69418ee4b1ba7091e9474db72da" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41557 } }
I20260926 14:39:26.598977 17316 catalog_manager.cc:5495] T f7437f733d4e43a2b6e6f1b999dca820 P 96e599d7ad594f7c930935d23381034e reported cstate change: term changed from 0 to 1, leader changed from <none> to 96e599d7ad594f7c930935d23381034e (127.16.90.193). New cstate: current_term: 1 leader_uuid: "96e599d7ad594f7c930935d23381034e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e599d7ad594f7c930935d23381034e" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9c2816961c34c49b60796043afe7deb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93b0d69418ee4b1ba7091e9474db72da" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41557 } health_report { overall_health: UNKNOWN } } }
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:39:26.622656 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.614220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.16.90.193:34885)
0926 14:39:26.614355 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:26.614360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:26.614505 (+ 145us) server_negotiation.cc:407] Connection header received
0926 14:39:26.614636 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:26.614638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:26.614684 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:26.614751 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:26.615581 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.616137 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.616961 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.617190 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.617279 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.619581 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:26.619594 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:26.619610 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:26.619634 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:26.621011 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.621404 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.621409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.621410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.621486 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.621743 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.621746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.621748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.622000 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:26.622192 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:26.622380 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 14:39:26.622492 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:39:26.647544 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.639700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.16.90.193:34885)
0926 14:39:26.639841 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:26.639846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:26.639967 (+ 121us) server_negotiation.cc:407] Connection header received
0926 14:39:26.640101 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:26.640104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:26.640147 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:26.640215 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:26.640884 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.641391 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.642214 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.642440 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.642530 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.644766 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:26.644769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:26.644771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:26.644795 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:26.645947 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.646384 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.646388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.646390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.646469 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.646704 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.646707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.646709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.646879 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:26.647010 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:26.647280 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 14:39:26.647392 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:39:26.670975 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:26.663424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.16.90.193:34885)
0926 14:39:26.663563 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:26.663567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:26.663698 (+ 131us) server_negotiation.cc:407] Connection header received
0926 14:39:26.663825 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:26.663828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:26.663874 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:26.663937 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:26.664598 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.665114 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.665914 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:26.666135 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:26.666221 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:26.668423 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:26.668426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:26.668428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:26.668451 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:26.669500 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.669869 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.669872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.669874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.669950 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:26.670191 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:26.670195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:26.670197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:26.670358 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:26.670498 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:26.670718 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 14:39:26.670825 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 14:39:26.672564 17422 tablet_service.cc:2908] Rejecting scan request for tablet f7437f733d4e43a2b6e6f1b999dca820: Uninitialized: clean time has not yet been initialized
I20260926 14:39:26.674535 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 17379
I20260926 14:39:26.680470 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 17513
I20260926 14:39:26.685272 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 17647
I20260926 14:39:26.691172 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 17285
2026-09-26T14:39:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2113 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17817](info): set up 2 sockets
Sep 26 14:39:26 dist-test-slave-x4sn krb5kdc[17817](info): commencing operation
krb5kdc: starting...
W20260926 14:39:27.632727 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:39:27 dist-test-slave-x4sn krb5kdc[17817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:39:27.749068 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:40645
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:40645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:27.854182 17830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:27.854425 17830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:27.854480 17830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:27.858441 17830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:27.858496 17830 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 14:39:27.858520 17830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:27.858541 17830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:27.858563 17830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:27.858592 17830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:27.863781 17830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:40645
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:40645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:27.864734 17830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:27.865691 17830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:27.873561 17835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:27.873615 17836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:27.873647 17838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:27.873800 17830 server_base.cc:1034] running on GCE node
I20260926 14:39:27.874395 17830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:27.875525 17830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:27.876706 17830 hybrid_clock.cc:648] HybridClock initialized: now 1790433567876687 us; error 29 us; skew 500 ppm
Sep 26 14:39:27 dist-test-slave-x4sn krb5kdc[17817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433567, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:27.884939 17830 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:39:27.885896 17830 webserver.cc:460] Webserver started at http://127.16.90.254:39429/ using document root <none> and password file <none>
I20260926 14:39:27.886447 17830 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:27.886518 17830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:27.886703 17830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:27.888945 17830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "839c3e0aa10b4b77b19a462d9477a5a6"
format_stamp: "Formatted at 2026-09-26 14:39:27 on dist-test-slave-x4sn"
server_key: "52675cb24d8759b19f0a64cb40481ca5"
server_key_iv: "234a65d29b67dc7f29e1df1556ebd0f7"
server_key_version: "encryptionkey@0"
I20260926 14:39:27.889496 17830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "839c3e0aa10b4b77b19a462d9477a5a6"
format_stamp: "Formatted at 2026-09-26 14:39:27 on dist-test-slave-x4sn"
server_key: "52675cb24d8759b19f0a64cb40481ca5"
server_key_iv: "234a65d29b67dc7f29e1df1556ebd0f7"
server_key_version: "encryptionkey@0"
I20260926 14:39:27.893461 17830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:39:27.895818 17845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:27.897013 17830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:27.897116 17830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "839c3e0aa10b4b77b19a462d9477a5a6"
format_stamp: "Formatted at 2026-09-26 14:39:27 on dist-test-slave-x4sn"
server_key: "52675cb24d8759b19f0a64cb40481ca5"
server_key_iv: "234a65d29b67dc7f29e1df1556ebd0f7"
server_key_version: "encryptionkey@0"
I20260926 14:39:27.897202 17830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:27.912190 17830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:27.914952 17830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:27.915102 17830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:27.922992 17830 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:40645
I20260926 14:39:27.923005 17907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:40645 every 8 connection(s)
I20260926 14:39:27.923991 17830 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:27.924649 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 17830
I20260926 14:39:27.924743 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:27.924983 16747 external_mini_cluster.cc:1428] Setting key 784d769867ad739bb5204ee16a62368f
I20260926 14:39:27.926857 17908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:27.933303 17908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6: Bootstrap starting.
Sep 26 14:39:27 dist-test-slave-x4sn krb5kdc[17817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:27.935767 17908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:27.937058 17908 log.cc:824] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:27.939013 17908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6: No bootstrap required, opened a new log
I20260926 14:39:27.939298 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:27.926369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.16.90.254:40645)
0926 14:39:27.926774 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:27.926789 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:27.926825 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:39:27.927486 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:27.927503 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:27.927886 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:27.928170 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:27.929034 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:27.930047 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:27.931166 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:27.931624 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:27.931819 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:27.934461 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:27.934467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:27.934481 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:27.934510 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:27.936308 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:27.936779 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:27.936784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:27.936789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:27.936903 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:27.937278 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:27.937282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:27.937284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:27.937728 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:27.937912 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:27.938216 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 14:39:27.938502 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":101,"threads_started":1}
I20260926 14:39:27.942358 17908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839c3e0aa10b4b77b19a462d9477a5a6" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40645 } }
I20260926 14:39:27.942579 17908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:27.942632 17908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839c3e0aa10b4b77b19a462d9477a5a6, State: Initialized, Role: FOLLOWER
I20260926 14:39:27.943040 17908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839c3e0aa10b4b77b19a462d9477a5a6" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40645 } }
I20260926 14:39:27.943167 17908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:27.943231 17908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:27.943321 17908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:27.944247 17908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839c3e0aa10b4b77b19a462d9477a5a6" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40645 } }
I20260926 14:39:27.944550 17908 leader_election.cc:302] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 839c3e0aa10b4b77b19a462d9477a5a6; no voters:
I20260926 14:39:27.944887 17908 leader_election.cc:288] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:27.945003 17913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:27.945231 17913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [term 1 LEADER]: Becoming Leader. State: Replica: 839c3e0aa10b4b77b19a462d9477a5a6, State: Running, Role: LEADER
I20260926 14:39:27.945559 17913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839c3e0aa10b4b77b19a462d9477a5a6" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40645 } }
I20260926 14:39:27.946033 17908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:27.947104 17914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "839c3e0aa10b4b77b19a462d9477a5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839c3e0aa10b4b77b19a462d9477a5a6" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40645 } } }
I20260926 14:39:27.947239 17914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:27.947604 17922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:27.947587 17915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 839c3e0aa10b4b77b19a462d9477a5a6. Latest consensus state: current_term: 1 leader_uuid: "839c3e0aa10b4b77b19a462d9477a5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839c3e0aa10b4b77b19a462d9477a5a6" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40645 } } }
I20260926 14:39:27.947745 17915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:27.950954 17922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:27.956468 17922 catalog_manager.cc:1347] Generated new cluster ID: 68201d93ef454aeda70054ae8fe51a76
I20260926 14:39:27.956532 17922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:27.962163 17922 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:27.963016 17922 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:27.971827 17922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6: Generated new TSK 0
I20260926 14:39:27.972535 17922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:39:28.006618 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40645
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:28.116626 17934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:28.116928 17934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:28.116992 17934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:28.121045 17934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:28.121116 17934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:28.121162 17934 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:39:28.121214 17934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:28.127630 17934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.90.254:40645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:28.128717 17934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:28.129712 17934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:28.138773 17939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:28.138854 17940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:28.138928 17942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:28.139001 17934 server_base.cc:1034] running on GCE node
I20260926 14:39:28.139374 17934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:28.140015 17934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:28.141170 17934 hybrid_clock.cc:648] HybridClock initialized: now 1790433568141162 us; error 37 us; skew 500 ppm
Sep 26 14:39:28 dist-test-slave-x4sn krb5kdc[17817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433568, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:28.143850 17934 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:39:28.144652 17934 webserver.cc:460] Webserver started at http://127.16.90.193:45603/ using document root <none> and password file <none>
I20260926 14:39:28.145165 17934 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:28.145220 17934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:28.145385 17934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:28.147460 17934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "5452be17bb15478d86b877fe90a8fb2f"
format_stamp: "Formatted at 2026-09-26 14:39:28 on dist-test-slave-x4sn"
server_key: "768a51716c98ef0bb3cace76d392867f"
server_key_iv: "0dd8d70aabe964236a9db67674a8154a"
server_key_version: "encryptionkey@0"
I20260926 14:39:28.148069 17934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "5452be17bb15478d86b877fe90a8fb2f"
format_stamp: "Formatted at 2026-09-26 14:39:28 on dist-test-slave-x4sn"
server_key: "768a51716c98ef0bb3cace76d392867f"
server_key_iv: "0dd8d70aabe964236a9db67674a8154a"
server_key_version: "encryptionkey@0"
I20260926 14:39:28.151537 17934 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:39:28.153860 17949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:28.155010 17934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:28.155115 17934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "5452be17bb15478d86b877fe90a8fb2f"
format_stamp: "Formatted at 2026-09-26 14:39:28 on dist-test-slave-x4sn"
server_key: "768a51716c98ef0bb3cace76d392867f"
server_key_iv: "0dd8d70aabe964236a9db67674a8154a"
server_key_version: "encryptionkey@0"
I20260926 14:39:28.155207 17934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:28.189934 17934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:28.193014 17934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:28.193192 17934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:28.194687 17934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:28.194739 17934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:28.194793 17934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:28.194824 17934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:39:28 dist-test-slave-x4sn krb5kdc[17817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433568, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:28.205030 17934 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:45099
I20260926 14:39:28.205066 18064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:45099 every 8 connection(s)
I20260926 14:39:28.205996 17934 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:39:28.208741 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.195554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:38533 (local address 127.16.90.254:40645)
0926 14:39:28.195705 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.195709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.196842 (+ 1133us) server_negotiation.cc:407] Connection header received
0926 14:39:28.197901 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.197905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.197966 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.198046 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:28.199653 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.200328 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.202039 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.202345 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.202429 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.205252 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:28.205256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:28.205259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:28.205287 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:28.206711 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.207256 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.207259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.207261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.207351 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.207660 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.207663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.207680 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.207874 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:28.207960 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.208438 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.208572 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:39:28.209474 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.196178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40645 (local address 127.16.90.193:38533)
0926 14:39:28.196692 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.196736 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.197662 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.198286 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.198296 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.198750 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:28.199468 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.199488 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.200435 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.200438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.201914 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.201926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.202502 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.202506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.202740 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.203383 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:28.203404 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:28.205096 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:28.206807 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:28.206813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:28.206828 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:28.207161 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:28.207433 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:28.207436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:28.207439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:28.207572 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:28.207970 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:28.207976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:28.208221 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.208382 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.208685 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":137,"threads_started":1}
I20260926 14:39:28.210692 18065 heartbeater.cc:347] Connected to a master server at 127.16.90.254:40645
I20260926 14:39:28.210830 18065 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:28.211333 18065 heartbeater.cc:511] Master 127.16.90.254:40645 requested a full tablet report, sending...
I20260926 14:39:28.212900 17861 ts_manager.cc:194] Registered new tserver with Master: 5452be17bb15478d86b877fe90a8fb2f (127.16.90.193:45099)
I20260926 14:39:28.212997 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 17934
I20260926 14:39:28.213097 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:39:28.213370 16747 external_mini_cluster.cc:1428] Setting key 5ca07b5b46b2c52199e0e45cf9b8ac55
I20260926 14:39:28.214413 17861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:38533
I20260926 14:39:28.220041 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.212826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40645 (local address 127.16.90.193:43737)
0926 14:39:28.213128 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.213140 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.213229 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.213443 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.213445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.213627 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:28.213743 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.213751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.214716 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.214720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.215507 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.215518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.216190 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.216193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.216283 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.216790 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:28.216800 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:28.217102 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:28.218736 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:28.218739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:28.218740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:28.218915 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:28.219187 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:28.219189 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:28.219191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:28.219253 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:28.219618 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:28.219619 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:28.219722 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.219800 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.219899 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":89,"threads_started":1}
I20260926 14:39:28.220146 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.212884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:43737 (local address 127.16.90.254:40645)
0926 14:39:28.212992 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.212997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.213207 (+ 210us) server_negotiation.cc:407] Connection header received
0926 14:39:28.213305 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.213308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.213350 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.213418 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:28.213862 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.214579 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.215633 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.215970 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.216068 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.217232 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:28.217235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:28.217238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:28.217260 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:28.218640 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.219014 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.219017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.219019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.219106 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.219327 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.219330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.219331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.219540 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:28.219640 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.219873 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.220002 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:39:28.238979 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40645
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:28.347790 18074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:28.348042 18074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:28.348085 18074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:28.351958 18074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:28.352011 18074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:28.352048 18074 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:39:28.352084 18074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:28.357163 18074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.90.254:40645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:28.358136 18074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:28.359139 18074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:28.368358 18082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:28.368494 18074 server_base.cc:1034] running on GCE node
W20260926 14:39:28.368366 18080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:28.368428 18079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:28.368942 18074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:28.369544 18074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:28.370695 18074 hybrid_clock.cc:648] HybridClock initialized: now 1790433568370670 us; error 39 us; skew 500 ppm
Sep 26 14:39:28 dist-test-slave-x4sn krb5kdc[17817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433568, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:28.373298 18074 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:39:28.374138 18074 webserver.cc:460] Webserver started at http://127.16.90.194:43733/ using document root <none> and password file <none>
I20260926 14:39:28.374677 18074 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:28.374740 18074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:28.374930 18074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:28.376950 18074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "a4421a4d7e274d40a5d9e7086134f781"
format_stamp: "Formatted at 2026-09-26 14:39:28 on dist-test-slave-x4sn"
server_key: "9c91784d6df4693205dd0297604c2894"
server_key_iv: "20b9294e071357c57d1384ae21913957"
server_key_version: "encryptionkey@0"
I20260926 14:39:28.377470 18074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "a4421a4d7e274d40a5d9e7086134f781"
format_stamp: "Formatted at 2026-09-26 14:39:28 on dist-test-slave-x4sn"
server_key: "9c91784d6df4693205dd0297604c2894"
server_key_iv: "20b9294e071357c57d1384ae21913957"
server_key_version: "encryptionkey@0"
I20260926 14:39:28.381048 18074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:28.383348 18089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:28.384611 18074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:39:28.384716 18074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "a4421a4d7e274d40a5d9e7086134f781"
format_stamp: "Formatted at 2026-09-26 14:39:28 on dist-test-slave-x4sn"
server_key: "9c91784d6df4693205dd0297604c2894"
server_key_iv: "20b9294e071357c57d1384ae21913957"
server_key_version: "encryptionkey@0"
I20260926 14:39:28.384805 18074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:28.398404 18074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:28.401592 18074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:28.401782 18074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:28.403236 18074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:28.403287 18074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:28.403339 18074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:28.403368 18074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:39:28 dist-test-slave-x4sn krb5kdc[17817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433568, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:28.413546 18074 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:37943
I20260926 14:39:28.413609 18204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:37943 every 8 connection(s)
I20260926 14:39:28.414415 18074 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:39:28.414938 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 18074
I20260926 14:39:28.415061 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:39:28.415381 16747 external_mini_cluster.cc:1428] Setting key b6bb526747de43182ff728bd4a6602be
I20260926 14:39:28.416558 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.404001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:52021 (local address 127.16.90.254:40645)
0926 14:39:28.404113 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.404116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.405238 (+ 1122us) server_negotiation.cc:407] Connection header received
0926 14:39:28.406284 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.406288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.406350 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.406445 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:28.407850 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.408385 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.409682 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.409940 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.410034 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.412923 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:28.412927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:28.412929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:28.412957 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:28.414353 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.414904 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.414908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.414910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.415003 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.415375 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.415379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.415381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.415596 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:28.415711 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.416277 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.416400 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:39:28.417502 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.404585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40645 (local address 127.16.90.194:52021)
0926 14:39:28.405080 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.405124 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.406023 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.406742 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.406751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.407072 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:28.407690 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.407712 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.408501 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.408505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.409551 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.409561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.410096 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.410100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.410300 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.410924 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:28.410942 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:28.412760 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:28.414458 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:28.414464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:28.414478 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:28.414784 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:28.415106 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:28.415109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:28.415111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:28.415281 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:28.415726 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:28.415732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:28.416022 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.416219 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.416528 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":138,"threads_started":1}
I20260926 14:39:28.418722 18205 heartbeater.cc:347] Connected to a master server at 127.16.90.254:40645
I20260926 14:39:28.418874 18205 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:28.419392 18205 heartbeater.cc:511] Master 127.16.90.254:40645 requested a full tablet report, sending...
I20260926 14:39:28.420450 17862 ts_manager.cc:194] Registered new tserver with Master: a4421a4d7e274d40a5d9e7086134f781 (127.16.90.194:37943)
I20260926 14:39:28.421168 17862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:52021
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
I20260926 14:39:28.430145 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.420858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:34199 (local address 127.16.90.254:40645)
0926 14:39:28.420974 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.420979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.423918 (+ 2939us) server_negotiation.cc:407] Connection header received
0926 14:39:28.424022 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.424024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.424066 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.424130 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:28.424533 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.425070 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.425892 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.426107 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.426198 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.427229 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:28.427233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:28.427235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:28.427258 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:28.428616 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.429024 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.429027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.429029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.429105 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.429346 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.429349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.429351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.429556 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:28.429644 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.429904 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.430007 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:39:28.430152 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.420798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40645 (local address 127.16.90.194:34199)
0926 14:39:28.423840 (+ 3042us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.423852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.423942 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.424153 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.424156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.424339 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:28.424431 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.424436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.425187 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.425191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.425789 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.425798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.426201 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.426205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.426291 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.426712 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:28.426723 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:28.427053 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:28.428752 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:28.428755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:28.428757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:28.428933 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:28.429190 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:28.429193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:28.429195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:28.429269 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:28.429645 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:28.429648 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:28.429739 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.429823 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.429957 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2971,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:39:28.442847 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40645
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:28.550784 18214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:28.551041 18214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:28.551100 18214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:28.555102 18214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:28.555166 18214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:28.555219 18214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:39:28.555274 18214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:28.560530 18214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.90.254:40645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:28.561554 18214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:28.562613 18214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:28.571830 18222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:28.571869 18219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:28.571851 18214 server_base.cc:1034] running on GCE node
W20260926 14:39:28.572120 18220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:28.572539 18214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:28.573134 18214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:28.574291 18214 hybrid_clock.cc:648] HybridClock initialized: now 1790433568574265 us; error 35 us; skew 500 ppm
Sep 26 14:39:28 dist-test-slave-x4sn krb5kdc[17817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433568, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:28.576930 18214 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:39:28.577770 18214 webserver.cc:460] Webserver started at http://127.16.90.195:33821/ using document root <none> and password file <none>
I20260926 14:39:28.578291 18214 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:28.578356 18214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:28.578544 18214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:28.580564 18214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "06f7e75c093845c097c4020d7eb2c057"
format_stamp: "Formatted at 2026-09-26 14:39:28 on dist-test-slave-x4sn"
server_key: "e46b9cf73070a0c55bda9eb265957b0e"
server_key_iv: "78594f478860137e99a901a2ac8e65f6"
server_key_version: "encryptionkey@0"
I20260926 14:39:28.581072 18214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "06f7e75c093845c097c4020d7eb2c057"
format_stamp: "Formatted at 2026-09-26 14:39:28 on dist-test-slave-x4sn"
server_key: "e46b9cf73070a0c55bda9eb265957b0e"
server_key_iv: "78594f478860137e99a901a2ac8e65f6"
server_key_version: "encryptionkey@0"
I20260926 14:39:28.584585 18214 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:28.586838 18229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:28.588020 18214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:28.588131 18214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "06f7e75c093845c097c4020d7eb2c057"
format_stamp: "Formatted at 2026-09-26 14:39:28 on dist-test-slave-x4sn"
server_key: "e46b9cf73070a0c55bda9eb265957b0e"
server_key_iv: "78594f478860137e99a901a2ac8e65f6"
server_key_version: "encryptionkey@0"
I20260926 14:39:28.588215 18214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:28.611334 18214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:28.614028 18214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:28.614195 18214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:28.615559 18214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:28.615607 18214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:28.615658 18214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:28.615716 18214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:39:28 dist-test-slave-x4sn krb5kdc[17817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433568, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:28.626099 18214 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:34411
I20260926 14:39:28.626207 18344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:34411 every 8 connection(s)
I20260926 14:39:28.626919 18214 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:28.628750 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.616895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40645 (local address 127.16.90.195:59475)
0926 14:39:28.617365 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.617407 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.618226 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.618743 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.618752 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.619219 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:28.619695 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.619714 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.620610 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.620613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.621530 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.621540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.622029 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.622033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.622219 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.622814 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:28.622835 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:28.624495 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:28.626167 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:28.626173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:28.626186 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:28.626495 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:28.626755 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:28.626758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:28.626760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:28.626883 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:28.627274 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:28.627280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:28.627539 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.627748 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.627994 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":129,"threads_started":1}
I20260926 14:39:28.628767 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 18214
I20260926 14:39:28.628870 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:39:28.629141 16747 external_mini_cluster.cc:1428] Setting key ce41b6dd1a5a8aef71f0b4984fbf5124
I20260926 14:39:28.629361 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.616361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:59475 (local address 127.16.90.254:40645)
0926 14:39:28.616472 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.616476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.617507 (+ 1031us) server_negotiation.cc:407] Connection header received
0926 14:39:28.618429 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.618433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.618491 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.618557 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:28.619856 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.620496 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.621653 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.621886 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.621974 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.624652 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:28.624663 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:28.624665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:28.624689 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:28.626042 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.626587 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.626591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.626593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.626677 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.626969 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.626972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.626974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.627176 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:28.627260 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.629112 (+ 1852us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.629227 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:39:28.630497 18345 heartbeater.cc:347] Connected to a master server at 127.16.90.254:40645
I20260926 14:39:28.630633 18345 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:28.631245 18345 heartbeater.cc:511] Master 127.16.90.254:40645 requested a full tablet report, sending...
I20260926 14:39:28.632102 17861 ts_manager.cc:194] Registered new tserver with Master: 06f7e75c093845c097c4020d7eb2c057 (127.16.90.195:34411)
I20260926 14:39:28.632702 17861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:59475
I20260926 14:39:28.638473 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.632501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:59443 (local address 127.16.90.254:40645)
0926 14:39:28.632615 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.632618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.632827 (+ 209us) server_negotiation.cc:407] Connection header received
0926 14:39:28.632929 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.632932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.632973 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.633035 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:28.633445 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.633930 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.634714 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.634928 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.635011 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.635967 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:28.635970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:28.635973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:28.635997 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:28.637008 (+ 1011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.637434 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.637437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.637439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.637514 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.637750 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.637754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.637755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.637917 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:28.637999 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.638235 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.638331 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:39:28.638484 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.632426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40645 (local address 127.16.90.195:59443)
0926 14:39:28.632760 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.632774 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.632862 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.633058 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.633061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.633252 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:28.633344 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.633349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.634058 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.634061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.634608 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.634616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.635022 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.635026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.635114 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.635519 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:28.635529 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:28.635850 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:28.637104 (+ 1254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:28.637107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:28.637109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:28.637347 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:28.637591 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:28.637594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:28.637595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:28.637674 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:28.638001 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:28.638004 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:28.638088 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.638179 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.638312 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260926 14:39:28.642658 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:39:28 dist-test-slave-x4sn krb5kdc[17817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433568, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:39:28 dist-test-slave-x4sn krb5kdc[17817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433568, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:28.667099 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.659412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.16.90.254:40645)
0926 14:39:28.659549 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.659552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.659564 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:39:28.659601 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.659603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.659642 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.659753 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:28.660400 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.660912 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.661760 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.661989 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.662224 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.664471 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:28.664474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:28.664476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:28.664503 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:28.665706 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.666050 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.666052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.666053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.666126 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:28.666334 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:28.666336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:28.666337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:28.666501 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:28.666600 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.666840 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.666942 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:39:28.670025 17861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:39:28.673113 17861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:28.686955 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.680772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:45099 (local address 127.0.0.1:47488)
0926 14:39:28.681202 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.681247 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.681360 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.682285 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.682291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.682318 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:28.682457 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.682463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.684088 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.684091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.685422 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.685433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.686476 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.686480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.686567 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.686593 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.686647 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.686778 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260926 14:39:28.687458 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.680851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:34411 (local address 127.0.0.1:52520)
0926 14:39:28.681558 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.681568 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.681640 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.682523 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.682526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.682541 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:28.682631 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.682638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.683635 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.683638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.685353 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.685365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.686411 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.686415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.686523 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.686567 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.686651 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.687327 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":563,"thread_start_us":107,"threads_started":1}
I20260926 14:39:28.687458 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.681470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:37943 (local address 127.0.0.1:55640)
0926 14:39:28.682183 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.682194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.682272 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.682712 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.682715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.682731 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:28.682831 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.682836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.684097 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.684099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.685917 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.685926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.687112 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.687116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.687205 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.687228 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.687290 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.687345 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":21,"thread_start_us":89,"threads_started":1}
I20260926 14:39:28.687810 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.681905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.16.90.194:37943)
0926 14:39:28.682311 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.682322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.682343 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:39:28.682409 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.682413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.682591 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.682737 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:28.682916 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.683994 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.686045 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.686819 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.686909 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.687053 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.687527 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.687620 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":99,"threads_started":1}
I20260926 14:39:28.687988 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.681840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.16.90.195:34411)
0926 14:39:28.682106 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.682117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.682139 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:39:28.682224 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.682229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.682409 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.682541 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:28.682756 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.683533 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.685490 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.686279 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.686778 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.686942 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.686996 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.687084 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":107,"threads_started":1}
I20260926 14:39:28.688241 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.680897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.16.90.193:45099)
0926 14:39:28.681592 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.681628 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.681654 (+ 26us) server_negotiation.cc:407] Connection header received
0926 14:39:28.681756 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.681869 (+ 113us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.682090 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.682601 (+ 511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:28.682736 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.683956 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.685571 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.686347 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.687799 (+ 1452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.687960 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.688007 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.688089 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":70,"threads_started":1}
I20260926 14:39:28.690589 17999 tablet_service.cc:1467] Processing CreateTablet for tablet 92004d1ffb3a4f6196ba85a63e67406a (DEFAULT_TABLE table=test-table [id=43c77830fdf146e9b40ec11ab514b298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:28.690610 18279 tablet_service.cc:1467] Processing CreateTablet for tablet 92004d1ffb3a4f6196ba85a63e67406a (DEFAULT_TABLE table=test-table [id=43c77830fdf146e9b40ec11ab514b298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:28.690610 18139 tablet_service.cc:1467] Processing CreateTablet for tablet 92004d1ffb3a4f6196ba85a63e67406a (DEFAULT_TABLE table=test-table [id=43c77830fdf146e9b40ec11ab514b298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:28.691615 18139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92004d1ffb3a4f6196ba85a63e67406a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:28.691622 18279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92004d1ffb3a4f6196ba85a63e67406a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:28.691624 17999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92004d1ffb3a4f6196ba85a63e67406a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:28.698045 18368 tablet_bootstrap.cc:492] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f: Bootstrap starting.
I20260926 14:39:28.699551 18370 tablet_bootstrap.cc:492] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057: Bootstrap starting.
I20260926 14:39:28.699551 18369 tablet_bootstrap.cc:492] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781: Bootstrap starting.
I20260926 14:39:28.700309 18368 tablet_bootstrap.cc:654] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:28.701081 18368 log.cc:824] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:28.701614 18370 tablet_bootstrap.cc:654] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:28.701614 18369 tablet_bootstrap.cc:654] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:28.702438 18370 log.cc:824] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:28.702438 18369 log.cc:824] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:28.703248 18368 tablet_bootstrap.cc:492] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f: No bootstrap required, opened a new log
I20260926 14:39:28.703428 18368 ts_tablet_manager.cc:1397] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:39:28.704348 18370 tablet_bootstrap.cc:492] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057: No bootstrap required, opened a new log
I20260926 14:39:28.704348 18369 tablet_bootstrap.cc:492] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781: No bootstrap required, opened a new log
I20260926 14:39:28.704524 18369 ts_tablet_manager.cc:1397] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:39:28.704524 18370 ts_tablet_manager.cc:1397] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:39:28.706358 18368 raft_consensus.cc:348] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } }
I20260926 14:39:28.706565 18368 raft_consensus.cc:374] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:28.706622 18368 raft_consensus.cc:729] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5452be17bb15478d86b877fe90a8fb2f, State: Initialized, Role: FOLLOWER
I20260926 14:39:28.707050 18368 consensus_queue.cc:260] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } }
I20260926 14:39:28.707722 18065 heartbeater.cc:503] Master 127.16.90.254:40645 was elected leader, sending a full tablet report...
I20260926 14:39:28.708052 18368 ts_tablet_manager.cc:1428] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:39:28.708233 18369 raft_consensus.cc:348] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } }
I20260926 14:39:28.708233 18370 raft_consensus.cc:348] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } }
I20260926 14:39:28.708472 18369 raft_consensus.cc:374] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:28.708472 18370 raft_consensus.cc:374] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:28.708524 18369 raft_consensus.cc:729] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4421a4d7e274d40a5d9e7086134f781, State: Initialized, Role: FOLLOWER
I20260926 14:39:28.708530 18370 raft_consensus.cc:729] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f7e75c093845c097c4020d7eb2c057, State: Initialized, Role: FOLLOWER
I20260926 14:39:28.709014 18369 consensus_queue.cc:260] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } }
I20260926 14:39:28.709044 18370 consensus_queue.cc:260] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } }
I20260926 14:39:28.709671 18345 heartbeater.cc:503] Master 127.16.90.254:40645 was elected leader, sending a full tablet report...
I20260926 14:39:28.709899 18205 heartbeater.cc:503] Master 127.16.90.254:40645 was elected leader, sending a full tablet report...
I20260926 14:39:28.710014 18370 ts_tablet_manager.cc:1428] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:39:28.710856 18369 ts_tablet_manager.cc:1428] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:39:28.780248 18375 raft_consensus.cc:482] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:28.780458 18375 raft_consensus.cc:504] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } }
I20260926 14:39:28.781661 18375 leader_election.cc:288] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5452be17bb15478d86b877fe90a8fb2f (127.16.90.193:45099), 06f7e75c093845c097c4020d7eb2c057 (127.16.90.195:34411)
I20260926 14:39:28.785185 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.781946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:45099 (local address 127.16.90.194:37285)
0926 14:39:28.782131 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.782146 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.782263 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.782496 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.782498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.782513 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:28.782620 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.782625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.783330 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.783333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.784156 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.784165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.784783 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.784787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.784888 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.784906 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.784970 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.785039 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 14:39:28.785185 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.782043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:37285 (local address 127.16.90.193:45099)
0926 14:39:28.782165 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.782170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.782210 (+ 40us) server_negotiation.cc:407] Connection header received
0926 14:39:28.782368 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.782370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.782411 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.782471 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:28.782722 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.783225 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.784278 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.784677 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.784782 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.784835 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.784998 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.785066 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:39:28.785831 18019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92004d1ffb3a4f6196ba85a63e67406a" candidate_uuid: "a4421a4d7e274d40a5d9e7086134f781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5452be17bb15478d86b877fe90a8fb2f" is_pre_election: true
I20260926 14:39:28.786123 18019 raft_consensus.cc:2454] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4421a4d7e274d40a5d9e7086134f781 in term 0.
I20260926 14:39:28.786615 18093 leader_election.cc:302] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5452be17bb15478d86b877fe90a8fb2f, a4421a4d7e274d40a5d9e7086134f781; no voters:
I20260926 14:39:28.786877 18375 raft_consensus.cc:2781] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:28.786983 18375 raft_consensus.cc:482] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:28.787039 18375 raft_consensus.cc:3037] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:28.788048 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.782113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:44369 (local address 127.16.90.195:34411)
0926 14:39:28.782241 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.782247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.782616 (+ 369us) server_negotiation.cc:407] Connection header received
0926 14:39:28.782740 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.782743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.782792 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.782861 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:28.783152 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.783687 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.787022 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.787451 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.787545 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.787582 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.787863 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.787951 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:39:28.788085 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.781945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:34411 (local address 127.16.90.194:44369)
0926 14:39:28.782519 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.782532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.782639 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.782876 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.782879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.782895 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:28.783032 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.783038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.783796 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.783799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.786874 (+ 3075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.786885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.787611 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.787615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.787737 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.787762 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.787851 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.787920 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":220,"thread_start_us":138,"threads_started":1}
I20260926 14:39:28.788630 18299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92004d1ffb3a4f6196ba85a63e67406a" candidate_uuid: "a4421a4d7e274d40a5d9e7086134f781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f7e75c093845c097c4020d7eb2c057" is_pre_election: true
I20260926 14:39:28.788812 18375 raft_consensus.cc:504] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } }
I20260926 14:39:28.788887 18299 raft_consensus.cc:2454] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4421a4d7e274d40a5d9e7086134f781 in term 0.
I20260926 14:39:28.789147 18375 leader_election.cc:288] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [CANDIDATE]: Term 1 election: Requested vote from peers 5452be17bb15478d86b877fe90a8fb2f (127.16.90.193:45099), 06f7e75c093845c097c4020d7eb2c057 (127.16.90.195:34411)
I20260926 14:39:28.789461 18019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92004d1ffb3a4f6196ba85a63e67406a" candidate_uuid: "a4421a4d7e274d40a5d9e7086134f781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5452be17bb15478d86b877fe90a8fb2f"
I20260926 14:39:28.789577 18299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92004d1ffb3a4f6196ba85a63e67406a" candidate_uuid: "a4421a4d7e274d40a5d9e7086134f781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f7e75c093845c097c4020d7eb2c057"
I20260926 14:39:28.789587 18019 raft_consensus.cc:3037] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:28.789695 18299 raft_consensus.cc:3037] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:28.790858 18019 raft_consensus.cc:2454] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4421a4d7e274d40a5d9e7086134f781 in term 1.
I20260926 14:39:28.791172 18093 leader_election.cc:302] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5452be17bb15478d86b877fe90a8fb2f, a4421a4d7e274d40a5d9e7086134f781; no voters:
I20260926 14:39:28.791219 18299 raft_consensus.cc:2454] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4421a4d7e274d40a5d9e7086134f781 in term 1.
I20260926 14:39:28.791356 18375 raft_consensus.cc:2781] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:28.791599 18375 raft_consensus.cc:686] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 1 LEADER]: Becoming Leader. State: Replica: a4421a4d7e274d40a5d9e7086134f781, State: Running, Role: LEADER
I20260926 14:39:28.791953 18375 consensus_queue.cc:237] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } }
I20260926 14:39:28.794965 17862 catalog_manager.cc:5495] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4421a4d7e274d40a5d9e7086134f781 (127.16.90.194). New cstate: current_term: 1 leader_uuid: "a4421a4d7e274d40a5d9e7086134f781" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } health_report { overall_health: HEALTHY } } }
I20260926 14:39:28.812881 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.809310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.16.90.194:37943)
0926 14:39:28.809425 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.809429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.809443 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:39:28.809647 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.809649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.809707 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.809787 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:28.810030 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.810532 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.811487 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.811748 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.811853 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.812138 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:28.812513 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:28.812610 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.812701 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.812753 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:39:28.818305 18299 raft_consensus.cc:1261] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057 [term 1 FOLLOWER]: Refusing update from remote peer a4421a4d7e274d40a5d9e7086134f781: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:28.818971 18375 consensus_queue.cc:1035] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:28.820027 18019 raft_consensus.cc:1261] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f [term 1 FOLLOWER]: Refusing update from remote peer a4421a4d7e274d40a5d9e7086134f781: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:28.820595 18375 consensus_queue.cc:1035] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:28.825500 18381 mvcc.cc:205] Tried to move back new op lower bound from 7333615897872523264 to 7333615897774415872. Current Snapshot: MvccSnapshot[applied={T|T < 7333615897872523264}]
I20260926 14:39:28.826692 18382 mvcc.cc:205] Tried to move back new op lower bound from 7333615897872523264 to 7333615897774415872. Current Snapshot: MvccSnapshot[applied={T|T < 7333615897872523264}]
I20260926 14:39:28.839145 17860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:28.839362 17860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:28.841771 17860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6: Sending DeleteTablet for 3 replicas of tablet 92004d1ffb3a4f6196ba85a63e67406a
I20260926 14:39:28.842510 18279 tablet_service.cc:1514] Processing DeleteTablet for tablet 92004d1ffb3a4f6196ba85a63e67406a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:28 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:52520
I20260926 14:39:28.842706 17999 tablet_service.cc:1514] Processing DeleteTablet for tablet 92004d1ffb3a4f6196ba85a63e67406a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:28 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:47488
I20260926 14:39:28.843081 18390 tablet_replica.cc:330] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057: stopping tablet replica
I20260926 14:39:28.843297 18139 tablet_service.cc:1514] Processing DeleteTablet for tablet 92004d1ffb3a4f6196ba85a63e67406a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:28 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:55640
I20260926 14:39:28.843389 18390 raft_consensus.cc:2229] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:28.843698 18390 raft_consensus.cc:2258] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:28.843657 18391 tablet_replica.cc:330] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f: stopping tablet replica
I20260926 14:39:28.843931 17860 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:39:28.844262 18391 raft_consensus.cc:2229] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 14:39:28.844307 17860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:28.844620 18391 raft_consensus.cc:2258] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:28.845033 18392 tablet_replica.cc:330] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781: stopping tablet replica
I20260926 14:39:28.845314 18390 ts_tablet_manager.cc:1905] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:28.845347 18392 raft_consensus.cc:2229] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:39:28.845835 18392 raft_consensus.cc:2258] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:28.847208 18392 ts_tablet_manager.cc:1905] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:28.847386 18391 ts_tablet_manager.cc:1905] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:28.849411 18279 tablet_service.cc:1467] Processing CreateTablet for tablet d8feb775c0e84ee6a7765dad65a2af8a (DEFAULT_TABLE table=test-table [id=b735294ba28846f5a58c48d6dbdc132d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:28.849639 18279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8feb775c0e84ee6a7765dad65a2af8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:28.849915 18392 ts_tablet_manager.cc:1918] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:39:28.849992 18392 log.cc:1196] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/92004d1ffb3a4f6196ba85a63e67406a
I20260926 14:39:28.849970 18390 ts_tablet_manager.cc:1918] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:39:28.850049 18390 log.cc:1196] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/92004d1ffb3a4f6196ba85a63e67406a
I20260926 14:39:28.850287 18392 ts_tablet_manager.cc:1939] T 92004d1ffb3a4f6196ba85a63e67406a P a4421a4d7e274d40a5d9e7086134f781: Deleting consensus metadata
I20260926 14:39:28.850327 18390 ts_tablet_manager.cc:1939] T 92004d1ffb3a4f6196ba85a63e67406a P 06f7e75c093845c097c4020d7eb2c057: Deleting consensus metadata
I20260926 14:39:28.851341 17849 catalog_manager.cc:4841] TS a4421a4d7e274d40a5d9e7086134f781 (127.16.90.194:37943): tablet 92004d1ffb3a4f6196ba85a63e67406a (table test-table [id=43c77830fdf146e9b40ec11ab514b298]) successfully deleted
I20260926 14:39:28.851366 17846 catalog_manager.cc:4841] TS 06f7e75c093845c097c4020d7eb2c057 (127.16.90.195:34411): tablet 92004d1ffb3a4f6196ba85a63e67406a (table test-table [id=43c77830fdf146e9b40ec11ab514b298]) successfully deleted
I20260926 14:39:28.851768 18370 tablet_bootstrap.cc:492] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057: Bootstrap starting.
I20260926 14:39:28.851848 17999 tablet_service.cc:1467] Processing CreateTablet for tablet d8feb775c0e84ee6a7765dad65a2af8a (DEFAULT_TABLE table=test-table [id=b735294ba28846f5a58c48d6dbdc132d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:28.852077 17999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8feb775c0e84ee6a7765dad65a2af8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:28.852617 18370 tablet_bootstrap.cc:654] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:28.853662 18391 ts_tablet_manager.cc:1918] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:39:28.853741 18391 log.cc:1196] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/wal/wals/92004d1ffb3a4f6196ba85a63e67406a
I20260926 14:39:28.854015 18391 ts_tablet_manager.cc:1939] T 92004d1ffb3a4f6196ba85a63e67406a P 5452be17bb15478d86b877fe90a8fb2f: Deleting consensus metadata
I20260926 14:39:28.854223 18370 tablet_bootstrap.cc:492] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057: No bootstrap required, opened a new log
I20260926 14:39:28.854300 18370 ts_tablet_manager.cc:1397] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:39:28.854296 18139 tablet_service.cc:1467] Processing CreateTablet for tablet d8feb775c0e84ee6a7765dad65a2af8a (DEFAULT_TABLE table=test-table [id=b735294ba28846f5a58c48d6dbdc132d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:28.854508 18139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8feb775c0e84ee6a7765dad65a2af8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:28.854703 18370 raft_consensus.cc:348] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:28.854800 18370 raft_consensus.cc:374] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:28.854827 18370 raft_consensus.cc:729] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f7e75c093845c097c4020d7eb2c057, State: Initialized, Role: FOLLOWER
I20260926 14:39:28.854936 18370 consensus_queue.cc:260] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:28.855163 18370 ts_tablet_manager.cc:1428] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:28.855271 17849 catalog_manager.cc:4841] TS 5452be17bb15478d86b877fe90a8fb2f (127.16.90.193:45099): tablet 92004d1ffb3a4f6196ba85a63e67406a (table test-table [id=43c77830fdf146e9b40ec11ab514b298]) successfully deleted
I20260926 14:39:28.855647 18368 tablet_bootstrap.cc:492] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f: Bootstrap starting.
I20260926 14:39:28.856539 18368 tablet_bootstrap.cc:654] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:28.856618 18369 tablet_bootstrap.cc:492] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781: Bootstrap starting.
I20260926 14:39:28.857494 18369 tablet_bootstrap.cc:654] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:28.857609 18368 tablet_bootstrap.cc:492] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f: No bootstrap required, opened a new log
I20260926 14:39:28.857703 18368 ts_tablet_manager.cc:1397] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 14:39:28.858323 18368 raft_consensus.cc:348] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:28.858426 18368 raft_consensus.cc:374] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:28.858458 18368 raft_consensus.cc:729] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5452be17bb15478d86b877fe90a8fb2f, State: Initialized, Role: FOLLOWER
I20260926 14:39:28.858564 18368 consensus_queue.cc:260] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:28.858657 18369 tablet_bootstrap.cc:492] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781: No bootstrap required, opened a new log
I20260926 14:39:28.858726 18369 ts_tablet_manager.cc:1397] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:28.858825 18368 ts_tablet_manager.cc:1428] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:39:28.859148 18369 raft_consensus.cc:348] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:28.859268 18369 raft_consensus.cc:374] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:28.859310 18369 raft_consensus.cc:729] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4421a4d7e274d40a5d9e7086134f781, State: Initialized, Role: FOLLOWER
I20260926 14:39:28.859431 18369 consensus_queue.cc:260] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:28.859699 18369 ts_tablet_manager.cc:1428] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260926 14:39:28.877822 18346 tablet.cc:2369] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:39:28.915750 18206 tablet.cc:2369] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:39:28.954933 18374 raft_consensus.cc:482] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:28.955071 18374 raft_consensus.cc:504] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:28.956168 18374 leader_election.cc:288] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4421a4d7e274d40a5d9e7086134f781 (127.16.90.194:37943), 06f7e75c093845c097c4020d7eb2c057 (127.16.90.195:34411)
W20260926 14:39:28.957317 18066 tablet.cc:2369] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:39:28.961004 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.956911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:33875 (local address 127.16.90.195:34411)
0926 14:39:28.957035 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.957040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.957263 (+ 223us) server_negotiation.cc:407] Connection header received
0926 14:39:28.957428 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.957431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.957475 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.957521 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:28.958125 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.958875 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.960038 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.960428 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.960540 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.960590 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.960817 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.960888 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:39:28.961022 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.956479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:34411 (local address 127.16.90.193:33875)
0926 14:39:28.957193 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.957204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.957321 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.957656 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.957659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.957708 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:28.957955 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.957966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.958966 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.958969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.959931 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.959940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.960560 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.960564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.960680 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.960707 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.960782 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.960852 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":125,"spinlock_wait_cycles":5888,"thread_start_us":99,"threads_started":1}
I20260926 14:39:28.961409 18299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8feb775c0e84ee6a7765dad65a2af8a" candidate_uuid: "5452be17bb15478d86b877fe90a8fb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f7e75c093845c097c4020d7eb2c057" is_pre_election: true
I20260926 14:39:28.961565 18299 raft_consensus.cc:2454] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5452be17bb15478d86b877fe90a8fb2f in term 0.
I20260926 14:39:28.961731 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.956414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:37943 (local address 127.16.90.193:54915)
0926 14:39:28.957130 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:28.957143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:28.957308 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:28.957656 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:28.957659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:28.957706 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:28.957882 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.957889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.958927 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.958931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.960293 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:28.960304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.961358 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.961362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:28.961438 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.961456 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:39:28.961519 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:39:28.961580 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":35,"spinlock_wait_cycles":52864,"thread_start_us":102,"threads_started":1}
I20260926 14:39:28.961820 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:28.956934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:54915 (local address 127.16.90.194:37943)
0926 14:39:28.957077 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:28.957082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:28.957228 (+ 146us) server_negotiation.cc:407] Connection header received
0926 14:39:28.957428 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:28.957431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:28.957475 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:28.957531 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:28.958023 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.958821 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.960425 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:28.960929 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:28.961033 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:28.961080 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:28.961627 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 14:39:28.961706 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:39:28.961992 17950 leader_election.cc:302] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f7e75c093845c097c4020d7eb2c057, 5452be17bb15478d86b877fe90a8fb2f; no voters:
I20260926 14:39:28.962237 18374 raft_consensus.cc:2781] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:28.962309 18374 raft_consensus.cc:482] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:28.962357 18374 raft_consensus.cc:3037] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:28.962343 18159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8feb775c0e84ee6a7765dad65a2af8a" candidate_uuid: "5452be17bb15478d86b877fe90a8fb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4421a4d7e274d40a5d9e7086134f781" is_pre_election: true
I20260926 14:39:28.962555 18159 raft_consensus.cc:2454] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5452be17bb15478d86b877fe90a8fb2f in term 0.
I20260926 14:39:28.963762 18374 raft_consensus.cc:504] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:28.964154 18374 leader_election.cc:288] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [CANDIDATE]: Term 1 election: Requested vote from peers a4421a4d7e274d40a5d9e7086134f781 (127.16.90.194:37943), 06f7e75c093845c097c4020d7eb2c057 (127.16.90.195:34411)
I20260926 14:39:28.964464 18159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8feb775c0e84ee6a7765dad65a2af8a" candidate_uuid: "5452be17bb15478d86b877fe90a8fb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4421a4d7e274d40a5d9e7086134f781"
I20260926 14:39:28.964470 18299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8feb775c0e84ee6a7765dad65a2af8a" candidate_uuid: "5452be17bb15478d86b877fe90a8fb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f7e75c093845c097c4020d7eb2c057"
I20260926 14:39:28.964565 18159 raft_consensus.cc:3037] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:28.964582 18299 raft_consensus.cc:3037] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:28.965468 18159 raft_consensus.cc:2454] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5452be17bb15478d86b877fe90a8fb2f in term 1.
I20260926 14:39:28.965723 17953 leader_election.cc:302] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5452be17bb15478d86b877fe90a8fb2f, a4421a4d7e274d40a5d9e7086134f781; no voters:
I20260926 14:39:28.965734 18299 raft_consensus.cc:2454] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5452be17bb15478d86b877fe90a8fb2f in term 1.
I20260926 14:39:28.966212 18374 raft_consensus.cc:2781] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:28.966457 18374 raft_consensus.cc:686] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 1 LEADER]: Becoming Leader. State: Replica: 5452be17bb15478d86b877fe90a8fb2f, State: Running, Role: LEADER
I20260926 14:39:28.966728 18374 consensus_queue.cc:237] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:28.969117 17862 catalog_manager.cc:5495] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5452be17bb15478d86b877fe90a8fb2f (127.16.90.193). New cstate: current_term: 1 leader_uuid: "5452be17bb15478d86b877fe90a8fb2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } health_report { overall_health: UNKNOWN } } }
Sep 26 14:39:29 dist-test-slave-x4sn krb5kdc[17817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433567, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:29.021380 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.013423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36438 (local address 127.16.90.254:40645)
0926 14:39:29.013593 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:29.013597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:29.013771 (+ 174us) server_negotiation.cc:407] Connection header received
0926 14:39:29.013876 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:29.013878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:29.013920 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:29.013989 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:29.014604 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.015118 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.015913 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.016136 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.016233 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.018298 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:29.018301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:29.018302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:29.018323 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:29.019706 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:29.020177 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:29.020180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:29.020182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:29.020254 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:29.020484 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:29.020487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:29.020489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:29.020704 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:29.020795 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:29.021083 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 14:39:29.021247 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":85}
I20260926 14:39:29.021381 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.013371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40645 (local address 127.0.0.1:36438)
0926 14:39:29.013668 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:29.013680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:29.013792 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:29.014018 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:29.014020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:29.014407 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:29.014502 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:29.014507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.015199 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.015201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:29.015783 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:29.015791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.016232 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.016234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:29.016287 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.016771 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:29.016781 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:29.018144 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:29.019805 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:29.019814 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:29.019822 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:29.020091 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:29.020329 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:29.020332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:29.020334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:29.020413 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:29.020810 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:29.020818 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:29.020931 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 14:39:29.021027 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 14:39:29.021193 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":127,"threads_started":1}
I20260926 14:39:29.025337 17862 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:36438:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 14:39:29.025988 17862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:29.030560 18139 tablet_service.cc:1467] Processing CreateTablet for tablet d99df45c1ff84f92afa1f4fe9e92a176 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a1119350caea464db6573fddd377120e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:39:29.030560 17999 tablet_service.cc:1467] Processing CreateTablet for tablet d99df45c1ff84f92afa1f4fe9e92a176 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a1119350caea464db6573fddd377120e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:39:29.030766 18279 tablet_service.cc:1467] Processing CreateTablet for tablet d99df45c1ff84f92afa1f4fe9e92a176 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a1119350caea464db6573fddd377120e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:39:29.030956 17999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d99df45c1ff84f92afa1f4fe9e92a176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:29.030956 18139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d99df45c1ff84f92afa1f4fe9e92a176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:29.031196 18279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d99df45c1ff84f92afa1f4fe9e92a176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:29.033695 18370 tablet_bootstrap.cc:492] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: Bootstrap starting.
I20260926 14:39:29.033804 18368 tablet_bootstrap.cc:492] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f: Bootstrap starting.
I20260926 14:39:29.034893 18370 tablet_bootstrap.cc:654] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:29.035197 18368 tablet_bootstrap.cc:654] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:29.035243 18369 tablet_bootstrap.cc:492] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781: Bootstrap starting.
I20260926 14:39:29.036248 18369 tablet_bootstrap.cc:654] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:29.037133 18370 tablet_bootstrap.cc:492] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: No bootstrap required, opened a new log
I20260926 14:39:29.037215 18368 tablet_bootstrap.cc:492] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f: No bootstrap required, opened a new log
I20260926 14:39:29.037225 18370 ts_tablet_manager.cc:1397] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 14:39:29.037321 18368 ts_tablet_manager.cc:1397] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 14:39:29.037358 18369 tablet_bootstrap.cc:492] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781: No bootstrap required, opened a new log
I20260926 14:39:29.037446 18369 ts_tablet_manager.cc:1397] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 14:39:29.037916 18370 raft_consensus.cc:348] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:29.037918 18368 raft_consensus.cc:348] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:29.038034 18370 raft_consensus.cc:374] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:29.038034 18368 raft_consensus.cc:374] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:29.037868 18369 raft_consensus.cc:348] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:29.038065 18370 raft_consensus.cc:729] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f7e75c093845c097c4020d7eb2c057, State: Initialized, Role: FOLLOWER
I20260926 14:39:29.038071 18368 raft_consensus.cc:729] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5452be17bb15478d86b877fe90a8fb2f, State: Initialized, Role: FOLLOWER
I20260926 14:39:29.038120 18369 raft_consensus.cc:374] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:29.038149 18369 raft_consensus.cc:729] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4421a4d7e274d40a5d9e7086134f781, State: Initialized, Role: FOLLOWER
I20260926 14:39:29.038198 18368 consensus_queue.cc:260] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:29.038259 18369 consensus_queue.cc:260] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:29.038254 18370 consensus_queue.cc:260] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:29.038494 18368 ts_tablet_manager.cc:1428] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:39:29.038441 18374 tablet_replica.cc:439] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } }
I20260926 14:39:29.038508 18369 ts_tablet_manager.cc:1428] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:39:29.038563 18374 tablet_replica.cc:442] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:29.038528 18376 tablet_replica.cc:439] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } }
I20260926 14:39:29.038721 18375 tablet_replica.cc:439] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } }
I20260926 14:39:29.038793 18376 tablet_replica.cc:442] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:29.038828 18375 tablet_replica.cc:442] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:29.039551 18370 ts_tablet_manager.cc:1428] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:29.085353 18376 raft_consensus.cc:482] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:29.085500 18376 raft_consensus.cc:504] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:29.086692 18376 leader_election.cc:288] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5452be17bb15478d86b877fe90a8fb2f (127.16.90.193:45099), a4421a4d7e274d40a5d9e7086134f781 (127.16.90.194:37943)
I20260926 14:39:29.090641 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.086903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:45099 (local address 127.16.90.195:35877)
0926 14:39:29.087019 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:29.087035 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:29.087121 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:29.087540 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:29.087543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:29.087564 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:29.087732 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:29.087739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.088562 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.088565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:29.089642 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:29.089652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.090257 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.090260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:29.090342 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.090366 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:39:29.090438 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 14:39:29.090507 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"spinlock_wait_cycles":29440}
I20260926 14:39:29.090647 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.086987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:35877 (local address 127.16.90.193:45099)
0926 14:39:29.087095 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:29.087099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:29.087116 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:39:29.087378 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:29.087381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:29.087429 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:29.087498 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:29.087825 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.088457 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.089757 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.090146 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.090267 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.090321 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:29.090459 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 14:39:29.090527 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:39:29.091114 18019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d99df45c1ff84f92afa1f4fe9e92a176" candidate_uuid: "06f7e75c093845c097c4020d7eb2c057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5452be17bb15478d86b877fe90a8fb2f" is_pre_election: true
I20260926 14:39:29.091261 18019 raft_consensus.cc:2454] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f7e75c093845c097c4020d7eb2c057 in term 0.
I20260926 14:39:29.091311 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.087170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:37943 (local address 127.16.90.195:43257)
0926 14:39:29.087469 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:29.087480 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:29.087627 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:29.087893 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:29.087894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:29.087909 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:29.088350 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:29.088358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.089070 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.089073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:29.090110 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:29.090118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.090887 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.090890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:29.090999 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.091020 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:39:29.091103 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 14:39:29.091165 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":4,"spinlock_wait_cycles":2176,"thread_start_us":94,"threads_started":1}
I20260926 14:39:29.091333 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.087263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:43257 (local address 127.16.90.194:37943)
0926 14:39:29.087374 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:29.087378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:29.087563 (+ 185us) server_negotiation.cc:407] Connection header received
0926 14:39:29.087761 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:29.087763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:29.087806 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:29.087881 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:29.088444 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.088966 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.090218 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.090785 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.090893 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.090938 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:29.091144 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 14:39:29.091224 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:39:29.091696 18233 leader_election.cc:302] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f7e75c093845c097c4020d7eb2c057, 5452be17bb15478d86b877fe90a8fb2f; no voters:
I20260926 14:39:29.091952 18376 raft_consensus.cc:2781] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:29.092013 18376 raft_consensus.cc:482] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:29.092049 18376 raft_consensus.cc:3037] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:29.092253 18159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d99df45c1ff84f92afa1f4fe9e92a176" candidate_uuid: "06f7e75c093845c097c4020d7eb2c057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4421a4d7e274d40a5d9e7086134f781" is_pre_election: true
I20260926 14:39:29.092381 18159 raft_consensus.cc:2454] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f7e75c093845c097c4020d7eb2c057 in term 0.
I20260926 14:39:29.093395 18376 raft_consensus.cc:504] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:29.093698 18376 leader_election.cc:288] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [CANDIDATE]: Term 1 election: Requested vote from peers 5452be17bb15478d86b877fe90a8fb2f (127.16.90.193:45099), a4421a4d7e274d40a5d9e7086134f781 (127.16.90.194:37943)
I20260926 14:39:29.094009 18019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d99df45c1ff84f92afa1f4fe9e92a176" candidate_uuid: "06f7e75c093845c097c4020d7eb2c057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5452be17bb15478d86b877fe90a8fb2f"
I20260926 14:39:29.094058 18159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d99df45c1ff84f92afa1f4fe9e92a176" candidate_uuid: "06f7e75c093845c097c4020d7eb2c057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4421a4d7e274d40a5d9e7086134f781"
I20260926 14:39:29.094108 18019 raft_consensus.cc:3037] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:29.094178 18159 raft_consensus.cc:3037] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:29.094949 18019 raft_consensus.cc:2454] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f7e75c093845c097c4020d7eb2c057 in term 1.
I20260926 14:39:29.095053 18159 raft_consensus.cc:2454] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f7e75c093845c097c4020d7eb2c057 in term 1.
I20260926 14:39:29.095191 18233 leader_election.cc:302] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f7e75c093845c097c4020d7eb2c057, 5452be17bb15478d86b877fe90a8fb2f; no voters:
I20260926 14:39:29.095333 18376 raft_consensus.cc:2781] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:29.095533 18376 raft_consensus.cc:686] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [term 1 LEADER]: Becoming Leader. State: Replica: 06f7e75c093845c097c4020d7eb2c057, State: Running, Role: LEADER
I20260926 14:39:29.095793 18376 consensus_queue.cc:237] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } }
I20260926 14:39:29.097046 18410 tablet_replica.cc:439] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: TxnStatusTablet state changed. Reason: New leader 06f7e75c093845c097c4020d7eb2c057. Latest consensus state: current_term: 1 leader_uuid: "06f7e75c093845c097c4020d7eb2c057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } }
I20260926 14:39:29.097148 18410 tablet_replica.cc:442] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:39:29.097613 18412 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 14:39:29.097779 18412 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 14:39:29.098309 17860 catalog_manager.cc:5495] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06f7e75c093845c097c4020d7eb2c057 (127.16.90.195). New cstate: current_term: 1 leader_uuid: "06f7e75c093845c097c4020d7eb2c057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } health_report { overall_health: HEALTHY } } }
I20260926 14:39:29.116355 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.112745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.16.90.195:34411)
0926 14:39:29.112847 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:29.112850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:29.112862 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:39:29.113026 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:29.113028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:29.113076 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:29.113152 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:29.113390 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.113862 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.115021 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.115230 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.115300 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.115586 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:29.116013 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:29.116096 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:29.116206 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 14:39:29.116250 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:39:29.116412 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.112612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:34411 (local address 127.0.0.1:52530)
0926 14:39:29.112768 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:29.112783 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:29.112885 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:29.113181 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:29.113184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:29.113192 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:39:29.113306 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:29.113312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.113950 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.113954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:29.114923 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:29.114933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.115325 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.115328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:29.115420 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.115471 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:39:29.116094 (+ 623us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:39:29.116107 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 14:39:29.116180 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:39:29.116274 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 14:39:29.119170 18019 raft_consensus.cc:1261] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f [term 1 FOLLOWER]: Refusing update from remote peer 06f7e75c093845c097c4020d7eb2c057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:29.119422 18159 raft_consensus.cc:1261] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781 [term 1 FOLLOWER]: Refusing update from remote peer 06f7e75c093845c097c4020d7eb2c057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:29.119625 18376 consensus_queue.cc:1035] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:29.119948 18410 consensus_queue.cc:1035] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:29.121429 18374 tablet_replica.cc:439] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f: TxnStatusTablet state changed. Reason: New leader 06f7e75c093845c097c4020d7eb2c057. Latest consensus state: current_term: 1 leader_uuid: "06f7e75c093845c097c4020d7eb2c057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } }
I20260926 14:39:29.121459 18375 tablet_replica.cc:439] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781: TxnStatusTablet state changed. Reason: New leader 06f7e75c093845c097c4020d7eb2c057. Latest consensus state: current_term: 1 leader_uuid: "06f7e75c093845c097c4020d7eb2c057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } }
I20260926 14:39:29.121537 18374 tablet_replica.cc:442] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:29.121546 18375 tablet_replica.cc:442] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:29.122995 18410 tablet_replica.cc:439] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06f7e75c093845c097c4020d7eb2c057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } }
I20260926 14:39:29.123090 18410 tablet_replica.cc:442] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:39:29.123948 18376 tablet_replica.cc:439] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06f7e75c093845c097c4020d7eb2c057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } }
I20260926 14:39:29.124044 18376 tablet_replica.cc:442] T d99df45c1ff84f92afa1f4fe9e92a176 P 06f7e75c093845c097c4020d7eb2c057: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:39:29.125290 18375 tablet_replica.cc:439] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06f7e75c093845c097c4020d7eb2c057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } }
I20260926 14:39:29.125392 18375 tablet_replica.cc:442] T d99df45c1ff84f92afa1f4fe9e92a176 P a4421a4d7e274d40a5d9e7086134f781: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:29.127094 18384 mvcc.cc:205] Tried to move back new op lower bound from 7333615899107532800 to 7333615899018604544. Current Snapshot: MvccSnapshot[applied={T|T < 7333615899107532800}]
I20260926 14:39:29.127283 18374 tablet_replica.cc:439] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06f7e75c093845c097c4020d7eb2c057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5452be17bb15478d86b877fe90a8fb2f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45099 } } peers { permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 } } peers { permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 } } }
I20260926 14:39:29.127374 18374 tablet_replica.cc:442] T d99df45c1ff84f92afa1f4fe9e92a176 P 5452be17bb15478d86b877fe90a8fb2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:29.133316 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.129324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.16.90.193:45099)
0926 14:39:29.129474 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:29.129478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:29.129493 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:39:29.129575 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:29.129578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:29.129636 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:29.129748 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:29.129988 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.130776 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.131706 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.132054 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.132154 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.132391 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:29.132902 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:29.133009 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:29.133112 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 14:39:29.133169 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 14:39:29.143281 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.140081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:60543 (local address 127.16.90.195:34411)
0926 14:39:29.140192 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:29.140195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:29.140441 (+ 246us) server_negotiation.cc:407] Connection header received
0926 14:39:29.140555 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:29.140558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:29.140603 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:29.140662 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:29.140900 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.141442 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.142300 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.142510 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.142596 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.142863 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:29.142930 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:29.142985 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:29.143089 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 14:39:29.143139 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:39:29.143530 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.139993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:34411 (local address 127.16.90.193:60543)
0926 14:39:29.140353 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:29.140365 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:29.140464 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:29.140677 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:29.140679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:29.140687 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:39:29.140792 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:29.140797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.141555 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.141559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:29.142192 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:29.142200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.142611 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.142615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:29.142700 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.142741 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:39:29.143001 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:39:29.143007 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:39:29.143065 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 14:39:29.143140 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":90,"threads_started":1}
I20260926 14:39:29.150143 18299 raft_consensus.cc:1261] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057 [term 1 FOLLOWER]: Refusing update from remote peer 5452be17bb15478d86b877fe90a8fb2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:29.150161 18159 raft_consensus.cc:1261] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781 [term 1 FOLLOWER]: Refusing update from remote peer 5452be17bb15478d86b877fe90a8fb2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:29.150668 18398 consensus_queue.cc:1035] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f7e75c093845c097c4020d7eb2c057" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:29.150843 18374 consensus_queue.cc:1035] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4421a4d7e274d40a5d9e7086134f781" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:29.173385 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.169535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:47185 (local address 127.16.90.193:45099)
0926 14:39:29.169644 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:29.169649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:29.169921 (+ 272us) server_negotiation.cc:407] Connection header received
0926 14:39:29.170042 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:29.170045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:29.170093 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:29.170166 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:29.170369 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.171136 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.172379 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.172619 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.172707 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.172980 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:29.173056 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:29.173112 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:29.173199 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 14:39:29.173246 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:39:29.173434 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.169393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:45099 (local address 127.16.90.195:47185)
0926 14:39:29.169827 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:29.169843 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:29.169959 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:29.170169 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:29.170171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:29.170191 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:39:29.170285 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:29.170289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.171254 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.171257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:29.172267 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:29.172276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.172717 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.172720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:29.172814 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.172866 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:39:29.173128 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:39:29.173134 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:39:29.173190 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:39:29.173276 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":124,"threads_started":1}
I20260926 14:39:29.201727 17860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:29.201920 17860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:29.204274 17860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 839c3e0aa10b4b77b19a462d9477a5a6: Sending DeleteTablet for 3 replicas of tablet d8feb775c0e84ee6a7765dad65a2af8a
I20260926 14:39:29.204913 18139 tablet_service.cc:1514] Processing DeleteTablet for tablet d8feb775c0e84ee6a7765dad65a2af8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:29 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:55640
I20260926 14:39:29.204975 18279 tablet_service.cc:1514] Processing DeleteTablet for tablet d8feb775c0e84ee6a7765dad65a2af8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:29 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:52520
I20260926 14:39:29.205101 18392 tablet_replica.cc:330] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781: stopping tablet replica
I20260926 14:39:29.205150 18390 tablet_replica.cc:330] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057: stopping tablet replica
I20260926 14:39:29.205220 18392 raft_consensus.cc:2229] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:29.205252 18390 raft_consensus.cc:2229] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:29.205329 18392 raft_consensus.cc:2258] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:29.205361 18390 raft_consensus.cc:2258] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:29.205893 17999 tablet_service.cc:1514] Processing DeleteTablet for tablet d8feb775c0e84ee6a7765dad65a2af8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:29 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:47488
I20260926 14:39:29.206050 18391 tablet_replica.cc:330] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f: stopping tablet replica
I20260926 14:39:29.206131 18390 ts_tablet_manager.cc:1905] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:29.206142 18391 raft_consensus.cc:2229] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:39:29.206398 18391 raft_consensus.cc:2258] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:29.207408 18391 ts_tablet_manager.cc:1905] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:29.208961 18390 ts_tablet_manager.cc:1918] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:39:29.209024 18390 log.cc:1196] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/d8feb775c0e84ee6a7765dad65a2af8a
I20260926 14:39:29.209244 18390 ts_tablet_manager.cc:1939] T d8feb775c0e84ee6a7765dad65a2af8a P 06f7e75c093845c097c4020d7eb2c057: Deleting consensus metadata
I20260926 14:39:29.209967 18391 ts_tablet_manager.cc:1918] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:39:29.210031 18391 log.cc:1196] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-0/wal/wals/d8feb775c0e84ee6a7765dad65a2af8a
I20260926 14:39:29.210029 17846 catalog_manager.cc:4841] TS 06f7e75c093845c097c4020d7eb2c057 (127.16.90.195:34411): tablet d8feb775c0e84ee6a7765dad65a2af8a (table test-table [id=b735294ba28846f5a58c48d6dbdc132d]) successfully deleted
I20260926 14:39:29.210237 18391 ts_tablet_manager.cc:1939] T d8feb775c0e84ee6a7765dad65a2af8a P 5452be17bb15478d86b877fe90a8fb2f: Deleting consensus metadata
I20260926 14:39:29.210806 17849 catalog_manager.cc:4841] TS 5452be17bb15478d86b877fe90a8fb2f (127.16.90.193:45099): tablet d8feb775c0e84ee6a7765dad65a2af8a (table test-table [id=b735294ba28846f5a58c48d6dbdc132d]) successfully deleted
I20260926 14:39:29.211797 18392 ts_tablet_manager.cc:1905] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:29.214344 18392 ts_tablet_manager.cc:1918] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:39:29.214409 18392 log.cc:1196] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/d8feb775c0e84ee6a7765dad65a2af8a
I20260926 14:39:29.214633 18392 ts_tablet_manager.cc:1939] T d8feb775c0e84ee6a7765dad65a2af8a P a4421a4d7e274d40a5d9e7086134f781: Deleting consensus metadata
I20260926 14:39:29.215435 17849 catalog_manager.cc:4841] TS a4421a4d7e274d40a5d9e7086134f781 (127.16.90.194:37943): tablet d8feb775c0e84ee6a7765dad65a2af8a (table test-table [id=b735294ba28846f5a58c48d6dbdc132d]) successfully deleted
Sep 26 14:39:29 dist-test-slave-x4sn krb5kdc[17817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433568, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:39:29.227433 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.215231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.16.90.193:45099)
0926 14:39:29.215340 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:29.215344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:29.215944 (+ 600us) server_negotiation.cc:407] Connection header received
0926 14:39:29.216065 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:29.216068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:29.216117 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:29.216193 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:29.216997 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.217735 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.218928 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.219294 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.219386 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.223335 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:29.223348 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:29.223364 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:29.223394 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:29.225339 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:29.225851 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:29.225857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:29.225859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:29.225947 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:29.226236 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:29.226239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:29.226241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:29.226546 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:29.226726 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:29.227143 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 14:39:29.227270 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 14:39:29.229050 18059 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47508
I20260926 14:39:29.242070 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:29.232071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.16.90.254:40645)
0926 14:39:29.232236 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:29.232241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:29.233522 (+ 1281us) server_negotiation.cc:407] Connection header received
0926 14:39:29.233633 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:29.233636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:29.233693 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:29.233775 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:29.234529 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.235345 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.236456 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:29.236817 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:29.236918 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:29.238566 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:29.238570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:29.238573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:29.238604 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:29.240255 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:29.240728 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:29.240732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:29.240734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:29.240824 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:29.241121 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:29.241124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:29.241126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:29.241349 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:29.241474 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:29.241771 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 14:39:29.241898 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 14:39:29.242632 17860 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36440
I20260926 14:39:29.244158 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 17934
I20260926 14:39:29.251595 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 18074
I20260926 14:39:29.258133 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 18214
I20260926 14:39:29.264423 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 17830
2026-09-26T14:39:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2573 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:39:29 dist-test-slave-x4sn krb5kdc[18433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:39:29 dist-test-slave-x4sn krb5kdc[18433](info): set up 2 sockets
Sep 26 14:39:29 dist-test-slave-x4sn krb5kdc[18433](info): commencing operation
krb5kdc: starting...
W20260926 14:39:30.220840 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:39:30 dist-test-slave-x4sn krb5kdc[18433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:39:30.332408 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35617
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:35617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:30.437819 18446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:30.438114 18446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:30.438158 18446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:30.442435 18446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:30.442510 18446 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 14:39:30.442528 18446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:30.442545 18446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:30.442560 18446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:30.442574 18446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:30.448076 18446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:35617
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:30.449131 18446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:30.450163 18446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:30.458580 18451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:30.458649 18452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:30.458663 18454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:30.458770 18446 server_base.cc:1034] running on GCE node
I20260926 14:39:30.459465 18446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:30.460538 18446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:30.461709 18446 hybrid_clock.cc:648] HybridClock initialized: now 1790433570461695 us; error 28 us; skew 500 ppm
Sep 26 14:39:30 dist-test-slave-x4sn krb5kdc[18433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433570, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:30.464290 18446 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:39:30.465140 18446 webserver.cc:460] Webserver started at http://127.16.90.254:43885/ using document root <none> and password file <none>
I20260926 14:39:30.465648 18446 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:30.465706 18446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:30.465901 18446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:30.467890 18446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "f044fcb40db14a989d6e65a01990dad8"
format_stamp: "Formatted at 2026-09-26 14:39:30 on dist-test-slave-x4sn"
server_key: "4601363b42077b55bbefa040fa9ba9cc"
server_key_iv: "391138e5ddcb8892bfebd134dd3df007"
server_key_version: "encryptionkey@0"
I20260926 14:39:30.468431 18446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "f044fcb40db14a989d6e65a01990dad8"
format_stamp: "Formatted at 2026-09-26 14:39:30 on dist-test-slave-x4sn"
server_key: "4601363b42077b55bbefa040fa9ba9cc"
server_key_iv: "391138e5ddcb8892bfebd134dd3df007"
server_key_version: "encryptionkey@0"
I20260926 14:39:30.472112 18446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:39:30.474355 18461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:30.475567 18446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:30.475708 18446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "f044fcb40db14a989d6e65a01990dad8"
format_stamp: "Formatted at 2026-09-26 14:39:30 on dist-test-slave-x4sn"
server_key: "4601363b42077b55bbefa040fa9ba9cc"
server_key_iv: "391138e5ddcb8892bfebd134dd3df007"
server_key_version: "encryptionkey@0"
I20260926 14:39:30.475805 18446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:30.489399 18446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:30.492443 18446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:30.492625 18446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:30.500345 18446 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:35617
I20260926 14:39:30.500366 18523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:35617 every 8 connection(s)
I20260926 14:39:30.501264 18446 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:30.504067 18524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:30.507900 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 18446
I20260926 14:39:30.508008 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:30.508258 16747 external_mini_cluster.cc:1428] Setting key 6c2b1c11682d517f91c58a6ad0b183e6
I20260926 14:39:30.509918 18524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8: Bootstrap starting.
I20260926 14:39:30.512382 18524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:30.513077 18524 log.cc:824] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:30.515079 18524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8: No bootstrap required, opened a new log
Sep 26 14:39:30 dist-test-slave-x4sn krb5kdc[18433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:30.519104 18524 raft_consensus.cc:348] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f044fcb40db14a989d6e65a01990dad8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35617 } }
I20260926 14:39:30.519398 18524 raft_consensus.cc:374] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:30.519460 18524 raft_consensus.cc:729] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f044fcb40db14a989d6e65a01990dad8, State: Initialized, Role: FOLLOWER
I20260926 14:39:30.519985 18524 consensus_queue.cc:260] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f044fcb40db14a989d6e65a01990dad8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35617 } }
I20260926 14:39:30.520148 18524 raft_consensus.cc:388] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:30.520226 18524 raft_consensus.cc:482] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:30.520325 18524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:30.521418 18524 raft_consensus.cc:504] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f044fcb40db14a989d6e65a01990dad8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35617 } }
I20260926 14:39:30.521792 18524 leader_election.cc:302] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f044fcb40db14a989d6e65a01990dad8; no voters:
I20260926 14:39:30.522140 18524 leader_election.cc:288] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:30.522246 18529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:30.522473 18529 raft_consensus.cc:686] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [term 1 LEADER]: Becoming Leader. State: Replica: f044fcb40db14a989d6e65a01990dad8, State: Running, Role: LEADER
I20260926 14:39:30.522667 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:30.509706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.16.90.254:35617)
0926 14:39:30.510091 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:30.510106 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:30.510142 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:39:30.510790 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:30.510807 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:30.511128 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:30.511409 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:30.512282 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.513306 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.514364 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.514823 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.514937 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:30.517503 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:30.517509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:30.517526 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:30.517557 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:30.519360 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:30.519910 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:30.519915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:30.519920 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:30.520058 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:30.520381 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:30.520385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:30.520387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:30.520895 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:30.521103 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:30.521339 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 14:39:30.521649 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":100,"threads_started":1}
I20260926 14:39:30.522773 18529 consensus_queue.cc:237] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f044fcb40db14a989d6e65a01990dad8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35617 } }
I20260926 14:39:30.523411 18524 sys_catalog.cc:564] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:30.523851 18529 sys_catalog.cc:455] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f044fcb40db14a989d6e65a01990dad8. Latest consensus state: current_term: 1 leader_uuid: "f044fcb40db14a989d6e65a01990dad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f044fcb40db14a989d6e65a01990dad8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35617 } } }
I20260926 14:39:30.523962 18529 sys_catalog.cc:458] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:30.524533 18530 sys_catalog.cc:455] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f044fcb40db14a989d6e65a01990dad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f044fcb40db14a989d6e65a01990dad8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35617 } } }
I20260926 14:39:30.524623 18530 sys_catalog.cc:458] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:30.524857 18533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:30.527520 18533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:30.532565 18533 catalog_manager.cc:1347] Generated new cluster ID: 6a07104a0b844bd29ed04144381e560e
I20260926 14:39:30.532634 18533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:30.549952 18533 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:30.551331 18533 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:30.565641 18533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8: Generated new TSK 0
I20260926 14:39:30.566525 18533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:39:30.599066 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35617
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:30.707976 18550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:30.708216 18550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:30.708263 18550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:30.712163 18550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:30.712216 18550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:30.712257 18550 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 14:39:30.712284 18550 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:39:30.712325 18550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:30.717446 18550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.90.254:35617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:30.718410 18550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:30.719358 18550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:30.728664 18558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:30.728713 18556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:30.728825 18550 server_base.cc:1034] running on GCE node
W20260926 14:39:30.728713 18555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:30.729256 18550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:30.729835 18550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:30.730995 18550 hybrid_clock.cc:648] HybridClock initialized: now 1790433570730977 us; error 28 us; skew 500 ppm
Sep 26 14:39:30 dist-test-slave-x4sn krb5kdc[18433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433570, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:30.733696 18550 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:39:30.734498 18550 webserver.cc:460] Webserver started at http://127.16.90.193:42949/ using document root <none> and password file <none>
I20260926 14:39:30.734992 18550 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:30.735047 18550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:30.735200 18550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:30.737123 18550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc"
format_stamp: "Formatted at 2026-09-26 14:39:30 on dist-test-slave-x4sn"
server_key: "3e6792f9baa88590e50bcce96babee00"
server_key_iv: "290603792cedb8eed054764521696ee7"
server_key_version: "encryptionkey@0"
I20260926 14:39:30.737591 18550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc"
format_stamp: "Formatted at 2026-09-26 14:39:30 on dist-test-slave-x4sn"
server_key: "3e6792f9baa88590e50bcce96babee00"
server_key_iv: "290603792cedb8eed054764521696ee7"
server_key_version: "encryptionkey@0"
I20260926 14:39:30.741047 18550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:30.743330 18565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:30.744580 18550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:30.744704 18550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc"
format_stamp: "Formatted at 2026-09-26 14:39:30 on dist-test-slave-x4sn"
server_key: "3e6792f9baa88590e50bcce96babee00"
server_key_iv: "290603792cedb8eed054764521696ee7"
server_key_version: "encryptionkey@0"
I20260926 14:39:30.744797 18550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:30.754499 18550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:30.757387 18550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:30.757543 18550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:30.758862 18550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:30.758906 18550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:30.758942 18550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:30.758970 18550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:39:30 dist-test-slave-x4sn krb5kdc[18433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433570, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:30.768702 18550 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:34091
I20260926 14:39:30.768751 18680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:34091 every 8 connection(s)
I20260926 14:39:30.769668 18550 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:39:30.772325 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:30.759742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:33637 (local address 127.16.90.254:35617)
0926 14:39:30.759881 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:30.759886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:30.760686 (+ 800us) server_negotiation.cc:407] Connection header received
0926 14:39:30.761751 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:30.761754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:30.761813 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:30.761892 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:30.763301 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.764152 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.765481 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.765802 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.765903 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:30.768962 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:30.768965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:30.768967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:30.769011 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:30.770474 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:30.770992 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:30.771024 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:30.771025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:30.771106 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:30.771357 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:30.771360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:30.771361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:30.771553 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:30.771627 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:30.772109 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 14:39:30.772210 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:39:30.773085 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:30.760078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35617 (local address 127.16.90.193:33637)
0926 14:39:30.760528 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:30.760576 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:30.761518 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:30.762072 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:30.762080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:30.762473 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:30.763140 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:30.763161 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.764272 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.764276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:30.765351 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:30.765362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.765957 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.765961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:30.766154 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:30.766754 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:30.766770 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:30.768746 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:30.770593 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:30.770598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:30.770613 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:30.770892 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:30.771168 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:30.771169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:30.771170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:30.771286 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:30.771629 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:30.771634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:30.771915 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 14:39:30.772066 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 14:39:30.772313 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
I20260926 14:39:30.774292 18681 heartbeater.cc:347] Connected to a master server at 127.16.90.254:35617
I20260926 14:39:30.774489 18681 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:30.775058 18681 heartbeater.cc:511] Master 127.16.90.254:35617 requested a full tablet report, sending...
I20260926 14:39:30.775228 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 18550
I20260926 14:39:30.775336 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:39:30.775617 16747 external_mini_cluster.cc:1428] Setting key 144db8d39082afbacf21e6c34181c42a
I20260926 14:39:30.776612 18477 ts_manager.cc:194] Registered new tserver with Master: 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091)
I20260926 14:39:30.777909 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:33637
I20260926 14:39:30.783583 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:30.776431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:38693 (local address 127.16.90.254:35617)
0926 14:39:30.776530 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:30.776534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:30.776755 (+ 221us) server_negotiation.cc:407] Connection header received
0926 14:39:30.776911 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:30.776914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:30.776958 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:30.777036 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:30.777486 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.778204 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.779357 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.779715 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.779809 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:30.780882 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:30.780885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:30.780887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:30.780907 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:30.782191 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:30.782620 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:30.782622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:30.782623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:30.782699 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:30.782937 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:30.782939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:30.782940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:30.783082 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:30.783147 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:30.783392 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 14:39:30.783474 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":40}
I20260926 14:39:30.783825 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:30.776395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35617 (local address 127.16.90.193:38693)
0926 14:39:30.776680 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:30.776693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:30.776816 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:30.777061 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:30.777063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:30.777260 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:30.777374 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:30.777381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.778330 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.778333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:30.779234 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:30.779244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.779830 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.779834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:30.779915 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:30.780362 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:30.780374 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:30.780759 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:30.782296 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:30.782300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:30.782302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:30.782536 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:30.782772 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:30.782775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:30.782776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:30.782865 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:30.783160 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:30.783162 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:30.783252 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 14:39:30.783346 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 14:39:30.783480 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:39:30.802623 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35617
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:30.911365 18690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:30.911625 18690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:30.911697 18690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:30.915606 18690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:30.915688 18690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:30.915741 18690 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 14:39:30.915762 18690 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:39:30.915793 18690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:30.920876 18690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.90.254:35617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:30.921847 18690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:30.922775 18690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:30.932063 18698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:30.932073 18695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:30.932107 18690 server_base.cc:1034] running on GCE node
W20260926 14:39:30.932075 18696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:30.932691 18690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:30.933280 18690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:30.934434 18690 hybrid_clock.cc:648] HybridClock initialized: now 1790433570934416 us; error 33 us; skew 500 ppm
Sep 26 14:39:30 dist-test-slave-x4sn krb5kdc[18433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433570, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:30.937041 18690 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:39:30.937862 18690 webserver.cc:460] Webserver started at http://127.16.90.194:41577/ using document root <none> and password file <none>
I20260926 14:39:30.938378 18690 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:30.938443 18690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:30.938622 18690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:30.940610 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "f576964c0138491e956d8198c2495d82"
format_stamp: "Formatted at 2026-09-26 14:39:30 on dist-test-slave-x4sn"
server_key: "6d61ef62d6fd0d01a3dc0339cc0139c6"
server_key_iv: "3e70f8a6b1092eb987ea3c079ce57768"
server_key_version: "encryptionkey@0"
I20260926 14:39:30.941124 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "f576964c0138491e956d8198c2495d82"
format_stamp: "Formatted at 2026-09-26 14:39:30 on dist-test-slave-x4sn"
server_key: "6d61ef62d6fd0d01a3dc0339cc0139c6"
server_key_iv: "3e70f8a6b1092eb987ea3c079ce57768"
server_key_version: "encryptionkey@0"
I20260926 14:39:30.944756 18690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:30.946971 18705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:30.948195 18690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:30.948346 18690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "f576964c0138491e956d8198c2495d82"
format_stamp: "Formatted at 2026-09-26 14:39:30 on dist-test-slave-x4sn"
server_key: "6d61ef62d6fd0d01a3dc0339cc0139c6"
server_key_iv: "3e70f8a6b1092eb987ea3c079ce57768"
server_key_version: "encryptionkey@0"
I20260926 14:39:30.948442 18690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:30.975414 18690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:30.978535 18690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:30.978708 18690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:30.980279 18690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:30.980335 18690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:30.980391 18690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:30.980419 18690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:39:30 dist-test-slave-x4sn krb5kdc[18433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433570, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:30.990386 18690 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:37255
I20260926 14:39:30.990511 18820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:37255 every 8 connection(s)
I20260926 14:39:30.991313 18690 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:39:30.994313 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:30.981218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:59391 (local address 127.16.90.254:35617)
0926 14:39:30.981341 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:30.981345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:30.982393 (+ 1048us) server_negotiation.cc:407] Connection header received
0926 14:39:30.983490 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:30.983494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:30.983551 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:30.983633 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:30.985238 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.986035 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.987716 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.987944 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.988035 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:30.990916 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:30.990919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:30.990921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:30.990947 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:30.992453 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:30.992940 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:30.992943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:30.992944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:30.993031 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:30.993328 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:30.993330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:30.993332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:30.993546 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:30.993625 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:30.994087 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 14:39:30.994192 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:39:30.995083 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:30.981688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35617 (local address 127.16.90.194:59391)
0926 14:39:30.982237 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:30.982281 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:30.983276 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:30.983874 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:30.983884 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:30.984348 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:30.985050 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:30.985072 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.986163 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.986168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:30.987575 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:30.987585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:30.988092 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:30.988097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:30.988292 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:30.988978 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:30.989001 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:30.990750 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:30.992560 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:30.992564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:30.992575 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:30.992832 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:30.993107 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:30.993109 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:30.993110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:30.993234 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:30.993625 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:30.993629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:30.993858 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 14:39:30.994007 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 14:39:30.994249 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":147,"threads_started":1}
I20260926 14:39:30.996368 18821 heartbeater.cc:347] Connected to a master server at 127.16.90.254:35617
I20260926 14:39:30.996495 18821 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:30.996973 18821 heartbeater.cc:511] Master 127.16.90.254:35617 requested a full tablet report, sending...
I20260926 14:39:30.997948 18478 ts_manager.cc:194] Registered new tserver with Master: f576964c0138491e956d8198c2495d82 (127.16.90.194:37255)
I20260926 14:39:30.998510 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 18690
I20260926 14:39:30.998623 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:39:30.998657 18478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:59391
I20260926 14:39:30.998929 16747 external_mini_cluster.cc:1428] Setting key 474bc548fcd7272b89f62913e62b13ec
I20260926 14:39:31.006099 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:30.998385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:40333 (local address 127.16.90.254:35617)
0926 14:39:30.999330 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:30.999334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:30.999569 (+ 235us) server_negotiation.cc:407] Connection header received
0926 14:39:30.999731 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:30.999734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:30.999783 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:30.999867 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:31.000335 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.000841 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.001642 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.001932 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.002014 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.003095 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:31.003098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:31.003100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:31.003121 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:31.004511 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:31.004964 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:31.004966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:31.004968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:31.005041 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:31.005303 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:31.005305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:31.005306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:31.005533 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:31.005606 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.005878 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.005975 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867}
I20260926 14:39:31.006198 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:30.998252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35617 (local address 127.16.90.194:40333)
0926 14:39:30.999492 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:30.999504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:30.999595 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:30.999894 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:30.999896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:31.000108 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:31.000239 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.000247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.000955 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.000958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.001521 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.001528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.002032 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.002035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.002126 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.002581 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:31.002594 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:31.002977 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:31.004624 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:31.004628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:31.004630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:31.004872 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:31.005115 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:31.005118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:31.005120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:31.005225 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:31.005617 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:31.005620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:31.005728 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 14:39:31.005840 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 14:39:31.005991 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:39:31.026149 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35617
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:31.133525 18830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:31.133790 18830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:31.133855 18830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:31.137871 18830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:31.137929 18830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:31.137984 18830 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 14:39:31.138008 18830 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:39:31.138073 18830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:31.143334 18830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.90.254:35617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:31.144405 18830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:31.145452 18830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:31.154843 18838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:31.154860 18835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:31.154848 18836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:31.155162 18830 server_base.cc:1034] running on GCE node
I20260926 14:39:31.155656 18830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:31.156303 18830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:31.157460 18830 hybrid_clock.cc:648] HybridClock initialized: now 1790433571157439 us; error 35 us; skew 500 ppm
Sep 26 14:39:31 dist-test-slave-x4sn krb5kdc[18433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:31.160136 18830 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:39:31.161093 18830 webserver.cc:460] Webserver started at http://127.16.90.195:36143/ using document root <none> and password file <none>
I20260926 14:39:31.161604 18830 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:31.161659 18830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:31.161837 18830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:31.163844 18830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "1d3afaa5ae6f4b0495783d77239efbd9"
format_stamp: "Formatted at 2026-09-26 14:39:31 on dist-test-slave-x4sn"
server_key: "cff75e38e4db0d255a5fb5c34818f4ae"
server_key_iv: "c493a37bcab02a6129d6955bc97ec322"
server_key_version: "encryptionkey@0"
I20260926 14:39:31.164345 18830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "1d3afaa5ae6f4b0495783d77239efbd9"
format_stamp: "Formatted at 2026-09-26 14:39:31 on dist-test-slave-x4sn"
server_key: "cff75e38e4db0d255a5fb5c34818f4ae"
server_key_iv: "c493a37bcab02a6129d6955bc97ec322"
server_key_version: "encryptionkey@0"
I20260926 14:39:31.167956 18830 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:39:31.170446 18845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:31.171777 18830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:31.171887 18830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "1d3afaa5ae6f4b0495783d77239efbd9"
format_stamp: "Formatted at 2026-09-26 14:39:31 on dist-test-slave-x4sn"
server_key: "cff75e38e4db0d255a5fb5c34818f4ae"
server_key_iv: "c493a37bcab02a6129d6955bc97ec322"
server_key_version: "encryptionkey@0"
I20260926 14:39:31.171986 18830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:31.186405 18830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:31.189203 18830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:31.189375 18830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:31.190892 18830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:31.190943 18830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:31.190995 18830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:31.191025 18830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:39:31 dist-test-slave-x4sn krb5kdc[18433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:31.200403 18830 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:36907
I20260926 14:39:31.201386 18830 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:31.201960 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 18830
I20260926 14:39:31.202073 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:39:31.202414 16747 external_mini_cluster.cc:1428] Setting key e5dd7412cef1270f70759fe96232de84
I20260926 14:39:31.202178 18960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:36907 every 8 connection(s)
I20260926 14:39:31.204700 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.191517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:59507 (local address 127.16.90.254:35617)
0926 14:39:31.191689 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:31.191694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:31.192932 (+ 1238us) server_negotiation.cc:407] Connection header received
0926 14:39:31.193987 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:31.193990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:31.194048 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:31.194122 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:31.195541 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.196134 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.197763 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.197999 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.198092 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.200978 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:31.200992 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:31.200994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:31.201020 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:31.202618 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:31.203204 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:31.203208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:31.203209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:31.203303 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:31.203604 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:31.203607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:31.203609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:31.203862 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:31.203952 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.204421 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.204543 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 14:39:31.205451 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.192337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35617 (local address 127.16.90.195:59507)
0926 14:39:31.192777 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:31.192825 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:31.193768 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:31.194306 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:31.194313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:31.194771 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:31.195404 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.195423 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.196253 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.196257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.197635 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.197645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.198136 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.198140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.198314 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.198922 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:31.198945 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:31.200809 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:31.202742 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:31.202747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:31.202760 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:31.203080 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:31.203379 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:31.203382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:31.203384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:31.203514 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:31.203960 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:31.203966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:31.204197 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 14:39:31.204361 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 14:39:31.204671 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
I20260926 14:39:31.206540 18961 heartbeater.cc:347] Connected to a master server at 127.16.90.254:35617
I20260926 14:39:31.206677 18961 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:31.207131 18961 heartbeater.cc:511] Master 127.16.90.254:35617 requested a full tablet report, sending...
I20260926 14:39:31.208094 18477 ts_manager.cc:194] Registered new tserver with Master: 1d3afaa5ae6f4b0495783d77239efbd9 (127.16.90.195:36907)
I20260926 14:39:31.208709 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:59507
I20260926 14:39:31.214545 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.208670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:58707 (local address 127.16.90.254:35617)
0926 14:39:31.208778 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:31.208782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:31.208991 (+ 209us) server_negotiation.cc:407] Connection header received
0926 14:39:31.209097 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:31.209100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:31.209138 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:31.209200 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:31.209620 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.210112 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.210893 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.211101 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.211184 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.212139 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:31.212142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:31.212143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:31.212166 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:31.213157 (+ 991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:31.213514 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:31.213518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:31.213520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:31.213589 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:31.213825 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:31.213829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:31.213831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:31.213988 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:31.214069 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.214313 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.214411 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:39:31.214558 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.208560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35617 (local address 127.16.90.195:58707)
0926 14:39:31.208922 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:31.208937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:31.209029 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:31.209225 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:31.209227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:31.209408 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:31.209514 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.209520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.210230 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.210233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.210789 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.210796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.211202 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.211205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.211301 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.211733 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:31.211747 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:31.212016 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:31.213253 (+ 1237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:31.213256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:31.213258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:31.213431 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:31.213669 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:31.213672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:31.213673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:31.213748 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:31.214073 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:31.214076 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:31.214167 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 14:39:31.214251 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:39:31.214378 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":8,"thread_start_us":91,"threads_started":1}
I20260926 14:39:31.216125 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:39:31 dist-test-slave-x4sn krb5kdc[18433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:39:31 dist-test-slave-x4sn krb5kdc[18433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:31.239976 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.232929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.16.90.254:35617)
0926 14:39:31.233060 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:31.233063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:31.233074 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:39:31.233114 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:31.233116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:31.233152 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:31.233238 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:31.233867 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.234387 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.235115 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.235331 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.235399 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.237595 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:31.237598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:31.237600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:31.237624 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:31.238703 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:31.239044 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:31.239046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:31.239047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:31.239112 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:31.239295 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:31.239297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:31.239298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:31.239460 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:31.239552 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.239734 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.239828 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:39:31.242899 18477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:39:31.246033 18477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:31.259938 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.254022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:34091 (local address 127.0.0.1:36836)
0926 14:39:31.254858 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:31.254899 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:31.255012 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:31.255569 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:31.255575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:31.255596 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:31.255752 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.255759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.257090 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.257094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.258497 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.258507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.259376 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.259379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.259482 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.259519 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 14:39:31.259633 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 14:39:31.259787 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":41,"thread_start_us":69,"threads_started":1}
I20260926 14:39:31.259943 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.254641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.16.90.193:34091)
0926 14:39:31.255108 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:31.255123 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:31.255144 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:39:31.255212 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:31.255217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:31.255422 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:31.255570 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:31.255851 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.256975 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.258613 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.259254 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.259354 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.259516 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.259691 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.259787 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":68,"threads_started":1}
I20260926 14:39:31.260283 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.254270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.16.90.195:36907)
0926 14:39:31.255308 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:31.255320 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:31.255620 (+ 300us) server_negotiation.cc:407] Connection header received
0926 14:39:31.255758 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:31.255763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:31.255949 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:31.256101 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:31.256486 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.257325 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.258671 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.259306 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.259403 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.259560 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.260058 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.260152 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937,"thread_start_us":106,"threads_started":1}
I20260926 14:39:31.259974 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.254121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:36907 (local address 127.0.0.1:40334)
0926 14:39:31.255435 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:31.255448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:31.255532 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:31.256253 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:31.256256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:31.256271 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:31.256365 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.256371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.257440 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.257444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.258559 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.258570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.259620 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.259623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.259743 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.259765 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:39:31.259826 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:39:31.259882 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260926 14:39:31.261153 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.254328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:37255 (local address 127.0.0.1:43252)
0926 14:39:31.255810 (+ 1482us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:31.255823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:31.255907 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:31.256313 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:31.256315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:31.256328 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:31.256405 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.256411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.257477 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.257481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.258884 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.258894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.260080 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.260084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.260856 (+ 772us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.260884 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 14:39:31.260954 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 14:39:31.261027 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1424,"mutex_wait_us":168,"thread_start_us":71,"threads_started":1}
I20260926 14:39:31.261350 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.254829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.16.90.194:37255)
0926 14:39:31.255366 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:31.255377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:31.255894 (+ 517us) server_negotiation.cc:407] Connection header received
0926 14:39:31.256004 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:31.256009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:31.256192 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:31.256330 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:31.256484 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.257369 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.259003 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.259788 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.259883 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.260015 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.260997 (+ 982us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.261081 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":105,"threads_started":1}
I20260926 14:39:31.263084 18895 tablet_service.cc:1467] Processing CreateTablet for tablet 1971d21a49b64f089b0ae8f1f8ba81b7 (DEFAULT_TABLE table=test-table [id=22cb5b84c73947b0925812d77214246c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:31.263084 18615 tablet_service.cc:1467] Processing CreateTablet for tablet 1971d21a49b64f089b0ae8f1f8ba81b7 (DEFAULT_TABLE table=test-table [id=22cb5b84c73947b0925812d77214246c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:31.264254 18895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1971d21a49b64f089b0ae8f1f8ba81b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:31.264249 18615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1971d21a49b64f089b0ae8f1f8ba81b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:31.265965 18755 tablet_service.cc:1467] Processing CreateTablet for tablet 1971d21a49b64f089b0ae8f1f8ba81b7 (DEFAULT_TABLE table=test-table [id=22cb5b84c73947b0925812d77214246c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:31.266799 18755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1971d21a49b64f089b0ae8f1f8ba81b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:31.271987 18984 tablet_bootstrap.cc:492] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9: Bootstrap starting.
I20260926 14:39:31.272393 18985 tablet_bootstrap.cc:492] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Bootstrap starting.
I20260926 14:39:31.273180 18986 tablet_bootstrap.cc:492] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82: Bootstrap starting.
I20260926 14:39:31.275034 18986 tablet_bootstrap.cc:654] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:31.275094 18984 tablet_bootstrap.cc:654] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:31.275830 18986 log.cc:824] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:31.275902 18984 log.cc:824] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:31.277276 18985 tablet_bootstrap.cc:654] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:31.277892 18984 tablet_bootstrap.cc:492] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9: No bootstrap required, opened a new log
I20260926 14:39:31.278039 18985 log.cc:824] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:31.278090 18984 ts_tablet_manager.cc:1397] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:39:31.278448 18986 tablet_bootstrap.cc:492] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82: No bootstrap required, opened a new log
I20260926 14:39:31.278626 18986 ts_tablet_manager.cc:1397] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260926 14:39:31.280521 18985 tablet_bootstrap.cc:492] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc: No bootstrap required, opened a new log
I20260926 14:39:31.280707 18985 ts_tablet_manager.cc:1397] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 14:39:31.281708 18986 raft_consensus.cc:348] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } }
I20260926 14:39:31.281936 18986 raft_consensus.cc:374] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:31.281985 18986 raft_consensus.cc:729] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f576964c0138491e956d8198c2495d82, State: Initialized, Role: FOLLOWER
I20260926 14:39:31.282166 18984 raft_consensus.cc:348] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } }
I20260926 14:39:31.282418 18984 raft_consensus.cc:374] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:31.282382 18986 consensus_queue.cc:260] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } }
I20260926 14:39:31.282474 18984 raft_consensus.cc:729] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3afaa5ae6f4b0495783d77239efbd9, State: Initialized, Role: FOLLOWER
I20260926 14:39:31.282894 18984 consensus_queue.cc:260] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } }
I20260926 14:39:31.283071 18821 heartbeater.cc:503] Master 127.16.90.254:35617 was elected leader, sending a full tablet report...
I20260926 14:39:31.283444 18986 ts_tablet_manager.cc:1428] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260926 14:39:31.283854 18984 ts_tablet_manager.cc:1428] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:39:31.284178 18961 heartbeater.cc:503] Master 127.16.90.254:35617 was elected leader, sending a full tablet report...
I20260926 14:39:31.284757 18985 raft_consensus.cc:348] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } }
I20260926 14:39:31.285006 18985 raft_consensus.cc:374] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:31.285072 18985 raft_consensus.cc:729] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ebc3c6a60f4ee0af190a7b6fc230bc, State: Initialized, Role: FOLLOWER
I20260926 14:39:31.285490 18985 consensus_queue.cc:260] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } }
I20260926 14:39:31.286202 18681 heartbeater.cc:503] Master 127.16.90.254:35617 was elected leader, sending a full tablet report...
I20260926 14:39:31.286623 18985 ts_tablet_manager.cc:1428] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:39:31.369256 18991 raft_consensus.cc:482] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:31.369472 18991 raft_consensus.cc:504] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } }
I20260926 14:39:31.370673 18991 leader_election.cc:288] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091), f576964c0138491e956d8198c2495d82 (127.16.90.194:37255)
I20260926 14:39:31.375069 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.371054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:54219 (local address 127.16.90.194:37255)
0926 14:39:31.371175 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:31.371179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:31.371194 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:39:31.371320 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:31.371322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:31.371375 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:31.371451 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:31.371832 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.372561 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.373950 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.374549 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.374691 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.374744 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.374904 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.374967 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:39:31.375093 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.370971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:37255 (local address 127.16.90.195:54219)
0926 14:39:31.371096 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:31.371112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:31.371209 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:31.371529 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:31.371532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:31.371558 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:31.371700 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.371706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.372701 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.372704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.373836 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.373844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.374667 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.374671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.374779 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.374805 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:39:31.374885 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 14:39:31.374959 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 14:39:31.375870 18775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1971d21a49b64f089b0ae8f1f8ba81b7" candidate_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f576964c0138491e956d8198c2495d82" is_pre_election: true
I20260926 14:39:31.375989 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.370975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:34091 (local address 127.16.90.195:47909)
0926 14:39:31.371658 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:31.371688 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:31.371777 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:31.372037 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:31.372039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:31.372058 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:31.372593 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.372602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.373628 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.373630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.374722 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.374731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.375602 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.375606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.375739 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.375763 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:39:31.375825 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 14:39:31.375874 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":312,"thread_start_us":151,"threads_started":1}
I20260926 14:39:31.376037 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.371008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:47909 (local address 127.16.90.193:34091)
0926 14:39:31.371139 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:31.371143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:31.371786 (+ 643us) server_negotiation.cc:407] Connection header received
0926 14:39:31.371855 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:31.371857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:31.371914 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:31.372022 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:31.372714 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.373527 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.374854 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.375468 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.375613 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.375658 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.375854 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.375926 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:39:31.376219 18775 raft_consensus.cc:2454] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d3afaa5ae6f4b0495783d77239efbd9 in term 0.
I20260926 14:39:31.376638 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1971d21a49b64f089b0ae8f1f8ba81b7" candidate_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" is_pre_election: true
I20260926 14:39:31.376729 18848 leader_election.cc:302] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3afaa5ae6f4b0495783d77239efbd9, f576964c0138491e956d8198c2495d82; no voters:
I20260926 14:39:31.376924 18635 raft_consensus.cc:2454] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d3afaa5ae6f4b0495783d77239efbd9 in term 0.
I20260926 14:39:31.377022 18991 raft_consensus.cc:2781] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:31.377137 18991 raft_consensus.cc:482] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:31.377190 18991 raft_consensus.cc:3037] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:31.378336 18991 raft_consensus.cc:504] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } }
I20260926 14:39:31.378672 18991 leader_election.cc:288] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [CANDIDATE]: Term 1 election: Requested vote from peers 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091), f576964c0138491e956d8198c2495d82 (127.16.90.194:37255)
I20260926 14:39:31.378979 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1971d21a49b64f089b0ae8f1f8ba81b7" candidate_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc"
I20260926 14:39:31.379052 18775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1971d21a49b64f089b0ae8f1f8ba81b7" candidate_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f576964c0138491e956d8198c2495d82"
I20260926 14:39:31.379099 18635 raft_consensus.cc:3037] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:31.379175 18775 raft_consensus.cc:3037] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:31.380182 18635 raft_consensus.cc:2454] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3afaa5ae6f4b0495783d77239efbd9 in term 1.
I20260926 14:39:31.380486 18846 leader_election.cc:302] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3afaa5ae6f4b0495783d77239efbd9, 35ebc3c6a60f4ee0af190a7b6fc230bc; no voters:
I20260926 14:39:31.380678 18991 raft_consensus.cc:2781] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:31.380698 18775 raft_consensus.cc:2454] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3afaa5ae6f4b0495783d77239efbd9 in term 1.
I20260926 14:39:31.380882 18991 raft_consensus.cc:686] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 LEADER]: Becoming Leader. State: Replica: 1d3afaa5ae6f4b0495783d77239efbd9, State: Running, Role: LEADER
I20260926 14:39:31.381244 18991 consensus_queue.cc:237] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } }
I20260926 14:39:31.384407 18477 catalog_manager.cc:5495] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d3afaa5ae6f4b0495783d77239efbd9 (127.16.90.195). New cstate: current_term: 1 leader_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } health_report { overall_health: UNKNOWN } } }
Sep 26 14:39:31 dist-test-slave-x4sn krb5kdc[18433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433570, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:31.433108 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.425364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.16.90.254:35617)
0926 14:39:31.425474 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:31.425479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:31.425704 (+ 225us) server_negotiation.cc:407] Connection header received
0926 14:39:31.425832 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:31.425835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:31.425871 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:31.425934 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:31.426589 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.427096 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.427876 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.428078 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.428160 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.430268 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:31.430271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:31.430273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:31.430292 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:31.431517 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:31.431990 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:31.431993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:31.431995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:31.432064 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:31.432298 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:31.432301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:31.432303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:31.432465 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:31.432548 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.432821 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.432969 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":38}
I20260926 14:39:31.433108 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.425341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35617 (local address 127.0.0.1:53688)
0926 14:39:31.425618 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:31.425630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:31.425751 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:31.425962 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:31.425965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:31.426386 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:31.426481 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.426486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.427191 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.427194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.427773 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.427782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.428170 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.428173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.428288 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.428813 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:31.428824 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:31.430145 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:31.431612 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:31.431622 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:31.431629 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:31.431891 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:31.432146 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:31.432149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:31.432151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:31.432226 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:31.432561 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:31.432568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:31.432674 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 14:39:31.432767 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 14:39:31.432921 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":25,"thread_start_us":96,"threads_started":1}
I20260926 14:39:31.436774 18477 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:53688:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 14:39:31.437521 18477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:31.442121 18615 tablet_service.cc:1467] Processing CreateTablet for tablet 810a78ab949148e89fd50f4a0eeb7113 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=970e5045bcdf4ac7aadcb17b8ea0d2d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:39:31.442152 18895 tablet_service.cc:1467] Processing CreateTablet for tablet 810a78ab949148e89fd50f4a0eeb7113 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=970e5045bcdf4ac7aadcb17b8ea0d2d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:39:31.442201 18755 tablet_service.cc:1467] Processing CreateTablet for tablet 810a78ab949148e89fd50f4a0eeb7113 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=970e5045bcdf4ac7aadcb17b8ea0d2d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:39:31.442871 18895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810a78ab949148e89fd50f4a0eeb7113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:31.442871 18615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810a78ab949148e89fd50f4a0eeb7113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:31.442878 18755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810a78ab949148e89fd50f4a0eeb7113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:31.445551 18986 tablet_bootstrap.cc:492] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82: Bootstrap starting.
I20260926 14:39:31.445658 18985 tablet_bootstrap.cc:492] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Bootstrap starting.
I20260926 14:39:31.445845 18984 tablet_bootstrap.cc:492] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: Bootstrap starting.
I20260926 14:39:31.446488 18986 tablet_bootstrap.cc:654] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:31.446602 18985 tablet_bootstrap.cc:654] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:31.446758 18984 tablet_bootstrap.cc:654] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:31.448038 18984 tablet_bootstrap.cc:492] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: No bootstrap required, opened a new log
I20260926 14:39:31.448128 18984 ts_tablet_manager.cc:1397] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:31.448146 18985 tablet_bootstrap.cc:492] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc: No bootstrap required, opened a new log
I20260926 14:39:31.448225 18985 ts_tablet_manager.cc:1397] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:39:31.448383 18986 tablet_bootstrap.cc:492] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82: No bootstrap required, opened a new log
I20260926 14:39:31.448486 18986 ts_tablet_manager.cc:1397] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:39:31.448567 18984 raft_consensus.cc:348] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.448689 18984 raft_consensus.cc:374] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:31.448726 18984 raft_consensus.cc:729] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3afaa5ae6f4b0495783d77239efbd9, State: Initialized, Role: FOLLOWER
I20260926 14:39:31.448719 18985 raft_consensus.cc:348] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.448839 18985 raft_consensus.cc:374] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:31.448880 18985 raft_consensus.cc:729] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ebc3c6a60f4ee0af190a7b6fc230bc, State: Initialized, Role: FOLLOWER
I20260926 14:39:31.448844 18984 consensus_queue.cc:260] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.448910 18986 raft_consensus.cc:348] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.449007 18986 raft_consensus.cc:374] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:31.449043 18986 raft_consensus.cc:729] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f576964c0138491e956d8198c2495d82, State: Initialized, Role: FOLLOWER
I20260926 14:39:31.449007 18985 consensus_queue.cc:260] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.449098 18984 ts_tablet_manager.cc:1428] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:31.449172 18986 consensus_queue.cc:260] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.449302 18985 ts_tablet_manager.cc:1428] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 14:39:31.449291 18991 tablet_replica.cc:439] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } }
I20260926 14:39:31.449407 18991 tablet_replica.cc:442] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:31.449438 18986 ts_tablet_manager.cc:1428] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:31.449542 18990 tablet_replica.cc:439] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } }
I20260926 14:39:31.449668 18990 tablet_replica.cc:442] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:31.450541 18992 tablet_replica.cc:439] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } }
I20260926 14:39:31.450696 18992 tablet_replica.cc:442] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 14:39:31.453169 18962 tablet.cc:2369] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:39:31.493124 18822 tablet.cc:2369] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:39:31.521021 18682 tablet.cc:2369] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:39:31.641547 18991 raft_consensus.cc:482] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:31.641705 18991 raft_consensus.cc:504] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.642119 18991 leader_election.cc:288] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091), f576964c0138491e956d8198c2495d82 (127.16.90.194:37255)
I20260926 14:39:31.642552 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810a78ab949148e89fd50f4a0eeb7113" candidate_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" is_pre_election: true
I20260926 14:39:31.642576 18775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810a78ab949148e89fd50f4a0eeb7113" candidate_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f576964c0138491e956d8198c2495d82" is_pre_election: true
I20260926 14:39:31.642696 18635 raft_consensus.cc:2454] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d3afaa5ae6f4b0495783d77239efbd9 in term 0.
I20260926 14:39:31.642725 18775 raft_consensus.cc:2454] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d3afaa5ae6f4b0495783d77239efbd9 in term 0.
I20260926 14:39:31.642982 18846 leader_election.cc:302] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3afaa5ae6f4b0495783d77239efbd9, 35ebc3c6a60f4ee0af190a7b6fc230bc; no voters:
I20260926 14:39:31.643159 18991 raft_consensus.cc:2781] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:31.643213 18991 raft_consensus.cc:482] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:31.643240 18991 raft_consensus.cc:3037] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:31.644205 18991 raft_consensus.cc:504] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.644495 18991 leader_election.cc:288] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [CANDIDATE]: Term 1 election: Requested vote from peers 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091), f576964c0138491e956d8198c2495d82 (127.16.90.194:37255)
I20260926 14:39:31.644819 18775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810a78ab949148e89fd50f4a0eeb7113" candidate_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f576964c0138491e956d8198c2495d82"
I20260926 14:39:31.644840 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810a78ab949148e89fd50f4a0eeb7113" candidate_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc"
I20260926 14:39:31.644918 18775 raft_consensus.cc:3037] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:31.644949 18635 raft_consensus.cc:3037] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:31.645848 18775 raft_consensus.cc:2454] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3afaa5ae6f4b0495783d77239efbd9 in term 1.
I20260926 14:39:31.645943 18635 raft_consensus.cc:2454] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3afaa5ae6f4b0495783d77239efbd9 in term 1.
I20260926 14:39:31.646106 18848 leader_election.cc:302] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3afaa5ae6f4b0495783d77239efbd9, f576964c0138491e956d8198c2495d82; no voters:
I20260926 14:39:31.646302 18991 raft_consensus.cc:2781] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:31.646373 18991 raft_consensus.cc:686] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 LEADER]: Becoming Leader. State: Replica: 1d3afaa5ae6f4b0495783d77239efbd9, State: Running, Role: LEADER
I20260926 14:39:31.646502 18991 consensus_queue.cc:237] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.647172 18994 tablet_replica.cc:439] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: TxnStatusTablet state changed. Reason: New leader 1d3afaa5ae6f4b0495783d77239efbd9. Latest consensus state: current_term: 1 leader_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } }
I20260926 14:39:31.647290 18994 tablet_replica.cc:442] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:39:31.647564 19006 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 14:39:31.647814 19006 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 14:39:31.648286 18478 catalog_manager.cc:5495] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d3afaa5ae6f4b0495783d77239efbd9 (127.16.90.195). New cstate: current_term: 1 leader_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } health_report { overall_health: HEALTHY } } }
I20260926 14:39:31.688762 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.684901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.16.90.195:36907)
0926 14:39:31.685015 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:31.685019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:31.685143 (+ 124us) server_negotiation.cc:407] Connection header received
0926 14:39:31.685329 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:31.685331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:31.685384 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:31.685463 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:31.685694 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.686246 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.687400 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.687600 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.687703 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.688012 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:31.688430 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:31.688512 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.688595 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.688644 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:39:31.688830 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.684814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:36907 (local address 127.0.0.1:40342)
0926 14:39:31.685077 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:31.685092 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:31.685204 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:31.685485 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:31.685488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:31.685495 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:39:31.685604 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.685610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.686345 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.686349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.687301 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.687313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.687721 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.687725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.687826 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.687879 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:39:31.688508 (+ 629us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:39:31.688519 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 14:39:31.688587 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:39:31.688688 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260926 14:39:31.693792 18775 raft_consensus.cc:1261] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Refusing update from remote peer 1d3afaa5ae6f4b0495783d77239efbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:31.693794 18635 raft_consensus.cc:1261] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Refusing update from remote peer 1d3afaa5ae6f4b0495783d77239efbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:31.694554 18994 consensus_queue.cc:1035] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:31.694885 18991 consensus_queue.cc:1035] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:31.698567 18990 tablet_replica.cc:439] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82: TxnStatusTablet state changed. Reason: New leader 1d3afaa5ae6f4b0495783d77239efbd9. Latest consensus state: current_term: 1 leader_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } }
I20260926 14:39:31.698702 18990 tablet_replica.cc:442] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:31.699815 18992 tablet_replica.cc:439] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc: TxnStatusTablet state changed. Reason: New leader 1d3afaa5ae6f4b0495783d77239efbd9. Latest consensus state: current_term: 1 leader_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } }
I20260926 14:39:31.699944 18992 tablet_replica.cc:442] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:31.701797 18991 tablet_replica.cc:439] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } }
I20260926 14:39:31.701913 18991 tablet_replica.cc:442] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:39:31.702456 19009 mvcc.cc:205] Tried to move back new op lower bound from 7333615909651259392 to 7333615909465288704. Current Snapshot: MvccSnapshot[applied={T|T < 7333615909651259392}]
I20260926 14:39:31.703840 18990 tablet_replica.cc:439] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } }
I20260926 14:39:31.703946 18990 tablet_replica.cc:442] T 810a78ab949148e89fd50f4a0eeb7113 P f576964c0138491e956d8198c2495d82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:31.704382 19010 mvcc.cc:205] Tried to move back new op lower bound from 7333615909651259392 to 7333615909465288704. Current Snapshot: MvccSnapshot[applied={T|T < 7333615909651259392}]
I20260926 14:39:31.705696 18994 tablet_replica.cc:439] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } }
I20260926 14:39:31.705809 18994 tablet_replica.cc:442] T 810a78ab949148e89fd50f4a0eeb7113 P 1d3afaa5ae6f4b0495783d77239efbd9: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:39:31.707080 18992 tablet_replica.cc:439] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } } }
I20260926 14:39:31.707194 18992 tablet_replica.cc:442] T 810a78ab949148e89fd50f4a0eeb7113 P 35ebc3c6a60f4ee0af190a7b6fc230bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:39:31.708403 19012 mvcc.cc:205] Tried to move back new op lower bound from 7333615909651259392 to 7333615909465288704. Current Snapshot: MvccSnapshot[applied={T|T < 7333615909651259392}]
I20260926 14:39:31.712956 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.710025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.16.90.195:36907)
0926 14:39:31.710129 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:31.710133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:31.710145 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:39:31.710198 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:31.710201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:31.710248 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:31.710330 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:31.710638 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.711176 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.712049 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.712258 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.712340 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.712559 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:31.712627 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:31.712688 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.712784 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.712833 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:39:31.723732 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.720248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:59575 (local address 127.16.90.195:36907)
0926 14:39:31.720360 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:31.720364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:31.720564 (+ 200us) server_negotiation.cc:407] Connection header received
0926 14:39:31.720689 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:31.720691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:31.720729 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:31.720784 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:31.721015 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.721485 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.722714 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.722912 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.722979 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.723303 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:31.723364 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:31.723417 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:31.723510 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 14:39:31.723573 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":42}
I20260926 14:39:31.723811 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:31.720217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:36907 (local address 127.16.90.195:59575)
0926 14:39:31.720482 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:31.720496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:31.720602 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:31.720803 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:31.720805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:31.720826 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:39:31.720938 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.720944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.721571 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.721574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.722590 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:31.722601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:31.723020 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:31.723024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:31.723115 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:31.723164 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:39:31.723438 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:39:31.723443 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:39:31.723504 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:39:31.723611 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":19,"thread_start_us":90,"threads_started":1}
I20260926 14:39:31.730087 18635 raft_consensus.cc:1261] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Refusing update from remote peer 1d3afaa5ae6f4b0495783d77239efbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:31.730115 18775 raft_consensus.cc:1261] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Refusing update from remote peer 1d3afaa5ae6f4b0495783d77239efbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:31.730446 18991 consensus_queue.cc:1035] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:31.730717 18994 consensus_queue.cc:1035] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:31.768265 18477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:31.768465 18477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:31.771013 18477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8: Sending DeleteTablet for 3 replicas of tablet 1971d21a49b64f089b0ae8f1f8ba81b7
I20260926 14:39:31.771761 18615 tablet_service.cc:1514] Processing DeleteTablet for tablet 1971d21a49b64f089b0ae8f1f8ba81b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:31 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:36836
I20260926 14:39:31.772284 18755 tablet_service.cc:1514] Processing DeleteTablet for tablet 1971d21a49b64f089b0ae8f1f8ba81b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:31 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:43252
I20260926 14:39:31.772817 19026 tablet_replica.cc:330] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82: stopping tablet replica
I20260926 14:39:31.773176 19026 raft_consensus.cc:2229] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:31.773440 19026 raft_consensus.cc:2258] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:31.773708 18895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1971d21a49b64f089b0ae8f1f8ba81b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:31 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:40334
I20260926 14:39:31.774147 18477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:39:31.774596 18477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:31.775074 19026 ts_tablet_manager.cc:1905] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:31.778173 19026 ts_tablet_manager.cc:1918] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:39:31.778328 19026 log.cc:1196] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/1971d21a49b64f089b0ae8f1f8ba81b7
I20260926 14:39:31.778703 19026 ts_tablet_manager.cc:1939] T 1971d21a49b64f089b0ae8f1f8ba81b7 P f576964c0138491e956d8198c2495d82: Deleting consensus metadata
I20260926 14:39:31.779825 18615 tablet_service.cc:1467] Processing CreateTablet for tablet 9c485e1198ae45e6b92346740568227c (DEFAULT_TABLE table=test-table [id=0c1f6fd0cbf2495b823ce05c692256f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:31.780125 18615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c485e1198ae45e6b92346740568227c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:31.782542 18985 tablet_bootstrap.cc:492] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc: Bootstrap starting.
I20260926 14:39:31.782727 18464 catalog_manager.cc:4841] TS f576964c0138491e956d8198c2495d82 (127.16.90.194:37255): tablet 1971d21a49b64f089b0ae8f1f8ba81b7 (table test-table [id=22cb5b84c73947b0925812d77214246c]) successfully deleted
I20260926 14:39:31.783288 18985 tablet_bootstrap.cc:654] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:31.783365 18755 tablet_service.cc:1467] Processing CreateTablet for tablet 9c485e1198ae45e6b92346740568227c (DEFAULT_TABLE table=test-table [id=0c1f6fd0cbf2495b823ce05c692256f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:31.783596 18755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c485e1198ae45e6b92346740568227c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:31.784459 18985 tablet_bootstrap.cc:492] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc: No bootstrap required, opened a new log
I20260926 14:39:31.784538 18985 ts_tablet_manager.cc:1397] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 14:39:31.784960 19025 tablet_replica.cc:330] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc: stopping tablet replica
I20260926 14:39:31.784922 18985 raft_consensus.cc:348] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.785022 18985 raft_consensus.cc:374] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:31.785050 18985 raft_consensus.cc:729] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ebc3c6a60f4ee0af190a7b6fc230bc, State: Initialized, Role: FOLLOWER
I20260926 14:39:31.785167 18985 consensus_queue.cc:260] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.785224 19025 raft_consensus.cc:2229] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:31.785429 18985 ts_tablet_manager.cc:1428] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:39:31.785430 19025 raft_consensus.cc:2258] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:31.786095 18986 tablet_bootstrap.cc:492] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82: Bootstrap starting.
I20260926 14:39:31.786697 18895 tablet_service.cc:1467] Processing CreateTablet for tablet 9c485e1198ae45e6b92346740568227c (DEFAULT_TABLE table=test-table [id=0c1f6fd0cbf2495b823ce05c692256f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:31.786881 18986 tablet_bootstrap.cc:654] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:31.786943 18895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c485e1198ae45e6b92346740568227c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:31.787909 18986 tablet_bootstrap.cc:492] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82: No bootstrap required, opened a new log
I20260926 14:39:31.787982 18986 ts_tablet_manager.cc:1397] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:39:31.788331 18986 raft_consensus.cc:348] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.788416 18986 raft_consensus.cc:374] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:31.788446 18986 raft_consensus.cc:729] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f576964c0138491e956d8198c2495d82, State: Initialized, Role: FOLLOWER
I20260926 14:39:31.788554 18986 consensus_queue.cc:260] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.788787 18986 ts_tablet_manager.cc:1428] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:31.789284 18984 tablet_bootstrap.cc:492] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9: Bootstrap starting.
I20260926 14:39:31.790251 18984 tablet_bootstrap.cc:654] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:31.791353 19027 tablet_replica.cc:330] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9: stopping tablet replica
I20260926 14:39:31.791546 19025 ts_tablet_manager.cc:1905] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:31.791625 19027 raft_consensus.cc:2229] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:39:31.791955 19027 raft_consensus.cc:2258] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:31.792400 18984 tablet_bootstrap.cc:492] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9: No bootstrap required, opened a new log
I20260926 14:39:31.792497 18984 ts_tablet_manager.cc:1397] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:39:31.792923 18984 raft_consensus.cc:348] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.793022 18984 raft_consensus.cc:374] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:31.793051 18984 raft_consensus.cc:729] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3afaa5ae6f4b0495783d77239efbd9, State: Initialized, Role: FOLLOWER
I20260926 14:39:31.793210 18984 consensus_queue.cc:260] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:31.793463 18984 ts_tablet_manager.cc:1428] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:31.794423 19025 ts_tablet_manager.cc:1918] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:39:31.794507 19025 log.cc:1196] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/wal/wals/1971d21a49b64f089b0ae8f1f8ba81b7
I20260926 14:39:31.793049 19027 ts_tablet_manager.cc:1905] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:31.794817 19025 ts_tablet_manager.cc:1939] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Deleting consensus metadata
I20260926 14:39:31.796065 18462 catalog_manager.cc:4841] TS 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091): tablet 1971d21a49b64f089b0ae8f1f8ba81b7 (table test-table [id=22cb5b84c73947b0925812d77214246c]) successfully deleted
I20260926 14:39:31.797406 19027 ts_tablet_manager.cc:1918] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:39:31.797498 19027 log.cc:1196] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/1971d21a49b64f089b0ae8f1f8ba81b7
I20260926 14:39:31.797755 19027 ts_tablet_manager.cc:1939] T 1971d21a49b64f089b0ae8f1f8ba81b7 P 1d3afaa5ae6f4b0495783d77239efbd9: Deleting consensus metadata
I20260926 14:39:31.798740 18465 catalog_manager.cc:4841] TS 1d3afaa5ae6f4b0495783d77239efbd9 (127.16.90.195:36907): tablet 1971d21a49b64f089b0ae8f1f8ba81b7 (table test-table [id=22cb5b84c73947b0925812d77214246c]) successfully deleted
I20260926 14:39:32.095353 18990 raft_consensus.cc:482] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:32.095497 18990 raft_consensus.cc:504] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:32.096675 18990 leader_election.cc:288] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091), 1d3afaa5ae6f4b0495783d77239efbd9 (127.16.90.195:36907)
I20260926 14:39:32.101799 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:32.097014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:60401 (local address 127.16.90.195:36907)
0926 14:39:32.097140 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:32.097144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:32.097430 (+ 286us) server_negotiation.cc:407] Connection header received
0926 14:39:32.097595 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:32.097598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:32.097645 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:32.097733 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:32.098067 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.098857 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.100584 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.101192 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.101313 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:32.101366 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:32.101584 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 14:39:32.101652 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:39:32.101799 19033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:32.097014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:33465 (local address 127.16.90.193:34091)
0926 14:39:32.097264 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:32.097268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:32.097379 (+ 111us) server_negotiation.cc:407] Connection header received
0926 14:39:32.097553 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:32.097557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:32.097610 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:32.097717 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:32.098068 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.098850 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.100585 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.101197 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.101313 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:32.101366 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:32.101584 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 14:39:32.101652 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260926 14:39:32.101866 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:32.096924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:36907 (local address 127.16.90.194:60401)
0926 14:39:32.097347 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:32.097358 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:32.097452 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:32.097741 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:32.097743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:32.097776 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:32.097960 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:32.097966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.098981 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.098984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:32.100445 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:32.100457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.101305 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.101309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:32.101445 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:32.101489 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 14:39:32.101564 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 14:39:32.101664 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":216,"spinlock_wait_cycles":4480,"thread_start_us":50,"threads_started":1}
I20260926 14:39:32.101873 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:32.096924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:34091 (local address 127.16.90.194:33465)
0926 14:39:32.097280 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:32.097296 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:32.097427 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:32.097735 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:32.097738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:32.097774 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:32.097956 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:32.097963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.098964 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.098968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:32.100445 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:32.100457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.101305 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.101309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:32.101468 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:32.101502 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 14:39:32.101564 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 14:39:32.101664 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":206,"spinlock_wait_cycles":17536,"thread_start_us":85,"threads_started":1}
I20260926 14:39:32.102397 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c485e1198ae45e6b92346740568227c" candidate_uuid: "f576964c0138491e956d8198c2495d82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" is_pre_election: true
I20260926 14:39:32.102399 18915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c485e1198ae45e6b92346740568227c" candidate_uuid: "f576964c0138491e956d8198c2495d82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" is_pre_election: true
I20260926 14:39:32.102561 18635 raft_consensus.cc:2454] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f576964c0138491e956d8198c2495d82 in term 0.
I20260926 14:39:32.102618 18915 raft_consensus.cc:2454] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f576964c0138491e956d8198c2495d82 in term 0.
I20260926 14:39:32.102952 18706 leader_election.cc:302] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ebc3c6a60f4ee0af190a7b6fc230bc, f576964c0138491e956d8198c2495d82; no voters:
I20260926 14:39:32.103202 18990 raft_consensus.cc:2781] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:32.103262 18990 raft_consensus.cc:482] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:32.103303 18990 raft_consensus.cc:3037] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:32.104452 18990 raft_consensus.cc:504] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:32.104773 18990 leader_election.cc:288] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [CANDIDATE]: Term 1 election: Requested vote from peers 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091), 1d3afaa5ae6f4b0495783d77239efbd9 (127.16.90.195:36907)
I20260926 14:39:32.105086 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c485e1198ae45e6b92346740568227c" candidate_uuid: "f576964c0138491e956d8198c2495d82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc"
I20260926 14:39:32.105085 18915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c485e1198ae45e6b92346740568227c" candidate_uuid: "f576964c0138491e956d8198c2495d82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3afaa5ae6f4b0495783d77239efbd9"
I20260926 14:39:32.105199 18915 raft_consensus.cc:3037] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:32.105233 18635 raft_consensus.cc:3037] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:32.106243 18635 raft_consensus.cc:2454] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f576964c0138491e956d8198c2495d82 in term 1.
I20260926 14:39:32.106371 18915 raft_consensus.cc:2454] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f576964c0138491e956d8198c2495d82 in term 1.
I20260926 14:39:32.106534 18706 leader_election.cc:302] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ebc3c6a60f4ee0af190a7b6fc230bc, f576964c0138491e956d8198c2495d82; no voters:
I20260926 14:39:32.106720 18990 raft_consensus.cc:2781] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:32.106851 18990 raft_consensus.cc:686] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 1 LEADER]: Becoming Leader. State: Replica: f576964c0138491e956d8198c2495d82, State: Running, Role: LEADER
I20260926 14:39:32.107057 18990 consensus_queue.cc:237] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:32.109208 18478 catalog_manager.cc:5495] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 reported cstate change: term changed from 0 to 1, leader changed from <none> to f576964c0138491e956d8198c2495d82 (127.16.90.194). New cstate: current_term: 1 leader_uuid: "f576964c0138491e956d8198c2495d82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } health_report { overall_health: UNKNOWN } } }
I20260926 14:39:32.165596 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:32.161862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.16.90.194:37255)
0926 14:39:32.162122 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:32.162128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:32.162140 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:39:32.162208 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:32.162211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:32.162267 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:32.162381 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:32.162548 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.163070 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.163873 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.164106 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.164182 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:32.164771 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:32.165243 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:32.165334 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:32.165377 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 14:39:32.165422 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":110,"threads_started":1}
I20260926 14:39:32.174969 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:32.172109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:49051 (local address 127.16.90.195:36907)
0926 14:39:32.172211 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:32.172214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:32.172391 (+ 177us) server_negotiation.cc:407] Connection header received
0926 14:39:32.172444 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:32.172446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:32.172489 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:32.172567 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:32.172745 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.173255 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.174040 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.174326 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.174406 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:32.174634 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:32.174702 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:32.174750 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:32.174822 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 14:39:32.174862 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:39:32.175005 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:32.172011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:36907 (local address 127.16.90.194:49051)
0926 14:39:32.172295 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:32.172307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:32.172387 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:32.172571 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:32.172573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:32.172578 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:39:32.172673 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:32.172677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.173339 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.173342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:32.173933 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:32.173940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.174408 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.174411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:32.174496 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:32.174537 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:39:32.174758 (+ 221us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:39:32.174761 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:39:32.174817 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:39:32.174885 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260926 14:39:32.180310 18635 raft_consensus.cc:1261] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Refusing update from remote peer f576964c0138491e956d8198c2495d82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:32.180490 18915 raft_consensus.cc:1261] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Refusing update from remote peer f576964c0138491e956d8198c2495d82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:32.180757 19034 consensus_queue.cc:1035] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:32.180910 18990 consensus_queue.cc:1035] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:32.196870 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:32.194015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:42507 (local address 127.16.90.194:37255)
0926 14:39:32.194123 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:32.194127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:32.194137 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:39:32.194170 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:32.194172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:32.194208 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:32.194269 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:32.194489 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.195042 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.195901 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.196179 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.196277 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:32.196495 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:32.196565 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:32.196623 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:32.196720 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 14:39:32.196771 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:39:32.196949 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:32.193633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:37255 (local address 127.16.90.195:42507)
0926 14:39:32.193741 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:32.193755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:32.193824 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:32.194296 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:32.194299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:32.194306 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:39:32.194401 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:32.194407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.195138 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.195142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:32.195780 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:32.195788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:32.196286 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:32.196289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:32.196373 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:32.196400 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:39:32.196645 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:39:32.196651 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:39:32.196708 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:39:32.196793 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 14:39:32.197611 19010 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 14:39:33.754902 18478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:33.755066 18478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:33.757768 18478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8: Sending DeleteTablet for 3 replicas of tablet 9c485e1198ae45e6b92346740568227c
I20260926 14:39:33.758358 18755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c485e1198ae45e6b92346740568227c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:33 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:43252
I20260926 14:39:33.758371 18615 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c485e1198ae45e6b92346740568227c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:33 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:36836
I20260926 14:39:33.758692 18895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c485e1198ae45e6b92346740568227c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:33 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:40334
I20260926 14:39:33.758759 19066 tablet_replica.cc:330] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc: stopping tablet replica
I20260926 14:39:33.758896 19066 raft_consensus.cc:2229] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:33.759023 19066 raft_consensus.cc:2258] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:33.759202 19068 tablet_replica.cc:330] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9: stopping tablet replica
I20260926 14:39:33.759335 19068 raft_consensus.cc:2229] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:33.759443 19068 raft_consensus.cc:2258] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:33.760066 19066 ts_tablet_manager.cc:1905] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:33.760223 19068 ts_tablet_manager.cc:1905] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:33.762049 19067 tablet_replica.cc:330] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82: stopping tablet replica
I20260926 14:39:33.762187 19067 raft_consensus.cc:2229] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:39:33.762467 19067 raft_consensus.cc:2258] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:33.762898 19066 ts_tablet_manager.cc:1918] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:39:33.762987 19066 log.cc:1196] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/wal/wals/9c485e1198ae45e6b92346740568227c
I20260926 14:39:33.762979 19067 ts_tablet_manager.cc:1905] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:33.763063 19068 ts_tablet_manager.cc:1918] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:39:33.763141 19068 log.cc:1196] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/9c485e1198ae45e6b92346740568227c
I20260926 14:39:33.763266 19066 ts_tablet_manager.cc:1939] T 9c485e1198ae45e6b92346740568227c P 35ebc3c6a60f4ee0af190a7b6fc230bc: Deleting consensus metadata
I20260926 14:39:33.763389 19068 ts_tablet_manager.cc:1939] T 9c485e1198ae45e6b92346740568227c P 1d3afaa5ae6f4b0495783d77239efbd9: Deleting consensus metadata
I20260926 14:39:33.764036 18462 catalog_manager.cc:4841] TS 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091): tablet 9c485e1198ae45e6b92346740568227c (table test-table [id=0c1f6fd0cbf2495b823ce05c692256f2]) successfully deleted
I20260926 14:39:33.764173 18465 catalog_manager.cc:4841] TS 1d3afaa5ae6f4b0495783d77239efbd9 (127.16.90.195:36907): tablet 9c485e1198ae45e6b92346740568227c (table test-table [id=0c1f6fd0cbf2495b823ce05c692256f2]) successfully deleted
I20260926 14:39:33.765529 19067 ts_tablet_manager.cc:1918] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:39:33.765605 19067 log.cc:1196] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/9c485e1198ae45e6b92346740568227c
I20260926 14:39:33.765856 19067 ts_tablet_manager.cc:1939] T 9c485e1198ae45e6b92346740568227c P f576964c0138491e956d8198c2495d82: Deleting consensus metadata
I20260926 14:39:33.766464 18464 catalog_manager.cc:4841] TS f576964c0138491e956d8198c2495d82 (127.16.90.194:37255): tablet 9c485e1198ae45e6b92346740568227c (table test-table [id=0c1f6fd0cbf2495b823ce05c692256f2]) successfully deleted
Sep 26 14:39:33 dist-test-slave-x4sn krb5kdc[18433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433573, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:39:33 dist-test-slave-x4sn krb5kdc[18433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433573, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:33.791553 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:33.779610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.16.90.254:35617)
0926 14:39:33.779915 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:33.779921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:33.780014 (+ 93us) server_negotiation.cc:407] Connection header received
0926 14:39:33.780140 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:33.780143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:33.780208 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:33.780293 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:33.780934 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:33.781491 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:33.782341 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:33.782560 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:33.782630 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:33.788153 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:33.788157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:33.788159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:33.788189 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:33.789877 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:33.790288 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:33.790292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:33.790294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:33.790390 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:33.790645 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:33.790649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:33.790651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:33.790889 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:33.791025 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:33.791262 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 14:39:33.791382 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":154,"threads_started":1}
I20260926 14:39:33.794698 18478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:39:33.795187 18478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:33.800202 18615 tablet_service.cc:1467] Processing CreateTablet for tablet afb4ceaa5ba7478089baf0c83b7c9a17 (DEFAULT_TABLE table=test-table [id=257c5fb09ec9446cb8ba2e2adadb3eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:33.800503 18615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb4ceaa5ba7478089baf0c83b7c9a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:33.800482 18755 tablet_service.cc:1467] Processing CreateTablet for tablet afb4ceaa5ba7478089baf0c83b7c9a17 (DEFAULT_TABLE table=test-table [id=257c5fb09ec9446cb8ba2e2adadb3eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:33.800544 18895 tablet_service.cc:1467] Processing CreateTablet for tablet afb4ceaa5ba7478089baf0c83b7c9a17 (DEFAULT_TABLE table=test-table [id=257c5fb09ec9446cb8ba2e2adadb3eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:33.800724 18755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb4ceaa5ba7478089baf0c83b7c9a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:33.800799 18895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb4ceaa5ba7478089baf0c83b7c9a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:33.802748 19077 tablet_bootstrap.cc:492] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9: Bootstrap starting.
I20260926 14:39:33.803470 19079 tablet_bootstrap.cc:492] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82: Bootstrap starting.
I20260926 14:39:33.803591 19078 tablet_bootstrap.cc:492] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Bootstrap starting.
I20260926 14:39:33.803649 19077 tablet_bootstrap.cc:654] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:33.804273 19079 tablet_bootstrap.cc:654] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:33.804507 19078 tablet_bootstrap.cc:654] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:33.805998 19078 tablet_bootstrap.cc:492] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc: No bootstrap required, opened a new log
I20260926 14:39:33.806067 19077 tablet_bootstrap.cc:492] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9: No bootstrap required, opened a new log
I20260926 14:39:33.806089 19078 ts_tablet_manager.cc:1397] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:39:33.806149 19077 ts_tablet_manager.cc:1397] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 14:39:33.806529 19077 raft_consensus.cc:348] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:33.806529 19078 raft_consensus.cc:348] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:33.806648 19078 raft_consensus.cc:374] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:33.806646 19077 raft_consensus.cc:374] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:33.806679 19078 raft_consensus.cc:729] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ebc3c6a60f4ee0af190a7b6fc230bc, State: Initialized, Role: FOLLOWER
I20260926 14:39:33.806689 19077 raft_consensus.cc:729] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3afaa5ae6f4b0495783d77239efbd9, State: Initialized, Role: FOLLOWER
I20260926 14:39:33.806820 19077 consensus_queue.cc:260] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:33.806840 19078 consensus_queue.cc:260] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:33.807200 19078 ts_tablet_manager.cc:1428] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:33.808419 19079 tablet_bootstrap.cc:492] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82: No bootstrap required, opened a new log
I20260926 14:39:33.808507 19079 ts_tablet_manager.cc:1397] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 14:39:33.810149 19077 ts_tablet_manager.cc:1428] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260926 14:39:33.810375 19079 raft_consensus.cc:348] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:33.810498 19079 raft_consensus.cc:374] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:33.810534 19079 raft_consensus.cc:729] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f576964c0138491e956d8198c2495d82, State: Initialized, Role: FOLLOWER
I20260926 14:39:33.810650 19079 consensus_queue.cc:260] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:33.811467 19079 ts_tablet_manager.cc:1428] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 14:39:34.081046 19044 raft_consensus.cc:482] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:34.081218 19044 raft_consensus.cc:504] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:34.081655 19044 leader_election.cc:288] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091), f576964c0138491e956d8198c2495d82 (127.16.90.194:37255)
I20260926 14:39:34.082243 18774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb4ceaa5ba7478089baf0c83b7c9a17" candidate_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f576964c0138491e956d8198c2495d82" is_pre_election: true
I20260926 14:39:34.082247 18634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb4ceaa5ba7478089baf0c83b7c9a17" candidate_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" is_pre_election: true
I20260926 14:39:34.082417 18774 raft_consensus.cc:2454] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d3afaa5ae6f4b0495783d77239efbd9 in term 0.
I20260926 14:39:34.082417 18634 raft_consensus.cc:2454] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d3afaa5ae6f4b0495783d77239efbd9 in term 0.
I20260926 14:39:34.082782 18848 leader_election.cc:302] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d3afaa5ae6f4b0495783d77239efbd9, 35ebc3c6a60f4ee0af190a7b6fc230bc, f576964c0138491e956d8198c2495d82; no voters:
I20260926 14:39:34.082959 19044 raft_consensus.cc:2781] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:34.083029 19044 raft_consensus.cc:482] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:34.083058 19044 raft_consensus.cc:3037] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:34.084095 19044 raft_consensus.cc:504] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:34.084399 19044 leader_election.cc:288] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [CANDIDATE]: Term 1 election: Requested vote from peers 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091), f576964c0138491e956d8198c2495d82 (127.16.90.194:37255)
I20260926 14:39:34.084853 18774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb4ceaa5ba7478089baf0c83b7c9a17" candidate_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f576964c0138491e956d8198c2495d82"
I20260926 14:39:34.084853 18634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb4ceaa5ba7478089baf0c83b7c9a17" candidate_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc"
I20260926 14:39:34.084987 18634 raft_consensus.cc:3037] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:34.084987 18774 raft_consensus.cc:3037] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:34.086129 18634 raft_consensus.cc:2454] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3afaa5ae6f4b0495783d77239efbd9 in term 1.
I20260926 14:39:34.086128 18774 raft_consensus.cc:2454] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3afaa5ae6f4b0495783d77239efbd9 in term 1.
I20260926 14:39:34.086419 18846 leader_election.cc:302] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3afaa5ae6f4b0495783d77239efbd9, 35ebc3c6a60f4ee0af190a7b6fc230bc; no voters:
I20260926 14:39:34.086587 19044 raft_consensus.cc:2781] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:34.086674 19044 raft_consensus.cc:686] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 LEADER]: Becoming Leader. State: Replica: 1d3afaa5ae6f4b0495783d77239efbd9, State: Running, Role: LEADER
I20260926 14:39:34.086808 19044 consensus_queue.cc:237] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:34.088442 18477 catalog_manager.cc:5495] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d3afaa5ae6f4b0495783d77239efbd9 (127.16.90.195). New cstate: current_term: 1 leader_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } health_report { overall_health: HEALTHY } } }
I20260926 14:39:34.108181 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:34.105234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.16.90.195:36907)
0926 14:39:34.105466 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:34.105478 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:34.105489 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:39:34.105526 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:34.105528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:34.105566 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:34.105631 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:34.105828 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:34.106371 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:34.107192 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:34.107437 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:34.107512 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:34.107752 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:34.107824 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:34.107877 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:34.108019 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 14:39:34.108063 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
I20260926 14:39:34.113956 18774 raft_consensus.cc:1261] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Refusing update from remote peer 1d3afaa5ae6f4b0495783d77239efbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:34.113956 18634 raft_consensus.cc:1261] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Refusing update from remote peer 1d3afaa5ae6f4b0495783d77239efbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:34.114315 19052 consensus_queue.cc:1035] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:34.114464 19047 consensus_queue.cc:1035] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:34.152956 18477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:34.153100 18477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:34.154803 18477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8: Sending DeleteTablet for 3 replicas of tablet afb4ceaa5ba7478089baf0c83b7c9a17
I20260926 14:39:34.155362 18615 tablet_service.cc:1514] Processing DeleteTablet for tablet afb4ceaa5ba7478089baf0c83b7c9a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:34 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:36836
I20260926 14:39:34.155370 18894 tablet_service.cc:1514] Processing DeleteTablet for tablet afb4ceaa5ba7478089baf0c83b7c9a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:34 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:40334
I20260926 14:39:34.155465 18755 tablet_service.cc:1514] Processing DeleteTablet for tablet afb4ceaa5ba7478089baf0c83b7c9a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:34 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:43252
I20260926 14:39:34.155532 19066 tablet_replica.cc:330] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc: stopping tablet replica
I20260926 14:39:34.155632 19066 raft_consensus.cc:2229] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:34.155654 19068 tablet_replica.cc:330] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9: stopping tablet replica
I20260926 14:39:34.155774 19067 tablet_replica.cc:330] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82: stopping tablet replica
I20260926 14:39:34.155776 19066 raft_consensus.cc:2258] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:34.155786 19068 raft_consensus.cc:2229] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:39:34.155871 19067 raft_consensus.cc:2229] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:34.155944 19068 raft_consensus.cc:2258] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:34.155956 19067 raft_consensus.cc:2258] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:34.156351 19066 ts_tablet_manager.cc:1905] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:34.156414 18477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:39:34.156782 19067 ts_tablet_manager.cc:1905] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 14:39:34.156780 18477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:34.157697 19068 ts_tablet_manager.cc:1905] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:34.158310 19066 ts_tablet_manager.cc:1918] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:39:34.158360 19066 log.cc:1196] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-0/wal/wals/afb4ceaa5ba7478089baf0c83b7c9a17
I20260926 14:39:34.158517 19066 ts_tablet_manager.cc:1939] T afb4ceaa5ba7478089baf0c83b7c9a17 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Deleting consensus metadata
I20260926 14:39:34.159040 18462 catalog_manager.cc:4841] TS 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091): tablet afb4ceaa5ba7478089baf0c83b7c9a17 (table test-table [id=257c5fb09ec9446cb8ba2e2adadb3eb0]) successfully deleted
I20260926 14:39:34.159111 19067 ts_tablet_manager.cc:1918] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:39:34.159173 19067 log.cc:1196] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/afb4ceaa5ba7478089baf0c83b7c9a17
I20260926 14:39:34.159397 19067 ts_tablet_manager.cc:1939] T afb4ceaa5ba7478089baf0c83b7c9a17 P f576964c0138491e956d8198c2495d82: Deleting consensus metadata
I20260926 14:39:34.160019 19068 ts_tablet_manager.cc:1918] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:39:34.160092 19068 log.cc:1196] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/afb4ceaa5ba7478089baf0c83b7c9a17
I20260926 14:39:34.160120 18464 catalog_manager.cc:4841] TS f576964c0138491e956d8198c2495d82 (127.16.90.194:37255): tablet afb4ceaa5ba7478089baf0c83b7c9a17 (table test-table [id=257c5fb09ec9446cb8ba2e2adadb3eb0]) successfully deleted
I20260926 14:39:34.160324 19068 ts_tablet_manager.cc:1939] T afb4ceaa5ba7478089baf0c83b7c9a17 P 1d3afaa5ae6f4b0495783d77239efbd9: Deleting consensus metadata
I20260926 14:39:34.160943 18465 catalog_manager.cc:4841] TS 1d3afaa5ae6f4b0495783d77239efbd9 (127.16.90.195:36907): tablet afb4ceaa5ba7478089baf0c83b7c9a17 (table test-table [id=257c5fb09ec9446cb8ba2e2adadb3eb0]) successfully deleted
I20260926 14:39:34.163059 18615 tablet_service.cc:1467] Processing CreateTablet for tablet 60417ca00a1040388b5c8922845d13f9 (DEFAULT_TABLE table=test-table [id=9ec8b3703c98452081de76171c886566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:34.163188 18894 tablet_service.cc:1467] Processing CreateTablet for tablet 60417ca00a1040388b5c8922845d13f9 (DEFAULT_TABLE table=test-table [id=9ec8b3703c98452081de76171c886566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:34.163220 18755 tablet_service.cc:1467] Processing CreateTablet for tablet 60417ca00a1040388b5c8922845d13f9 (DEFAULT_TABLE table=test-table [id=9ec8b3703c98452081de76171c886566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:34.163319 18615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60417ca00a1040388b5c8922845d13f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:34.163393 18894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60417ca00a1040388b5c8922845d13f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:34.163439 18755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60417ca00a1040388b5c8922845d13f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:34.164997 19079 tablet_bootstrap.cc:492] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82: Bootstrap starting.
I20260926 14:39:34.165410 19078 tablet_bootstrap.cc:492] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Bootstrap starting.
I20260926 14:39:34.165805 19079 tablet_bootstrap.cc:654] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:34.166217 19078 tablet_bootstrap.cc:654] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:34.166311 19077 tablet_bootstrap.cc:492] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9: Bootstrap starting.
I20260926 14:39:34.166778 19079 tablet_bootstrap.cc:492] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82: No bootstrap required, opened a new log
I20260926 14:39:34.166862 19079 ts_tablet_manager.cc:1397] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:34.167114 19077 tablet_bootstrap.cc:654] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:34.167217 19079 raft_consensus.cc:348] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:34.167299 19079 raft_consensus.cc:374] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:34.167317 19079 raft_consensus.cc:729] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f576964c0138491e956d8198c2495d82, State: Initialized, Role: FOLLOWER
I20260926 14:39:34.167364 19078 tablet_bootstrap.cc:492] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc: No bootstrap required, opened a new log
I20260926 14:39:34.167433 19078 ts_tablet_manager.cc:1397] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:34.167410 19079 consensus_queue.cc:260] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:34.167814 19078 raft_consensus.cc:348] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:34.167907 19078 raft_consensus.cc:374] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:34.167935 19078 raft_consensus.cc:729] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ebc3c6a60f4ee0af190a7b6fc230bc, State: Initialized, Role: FOLLOWER
I20260926 14:39:34.168048 19078 consensus_queue.cc:260] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:34.168102 19077 tablet_bootstrap.cc:492] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9: No bootstrap required, opened a new log
I20260926 14:39:34.168176 19077 ts_tablet_manager.cc:1397] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:39:34.167637 19079 ts_tablet_manager.cc:1428] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:34.168311 19078 ts_tablet_manager.cc:1428] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:34.168530 19077 raft_consensus.cc:348] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:34.168630 19077 raft_consensus.cc:374] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:34.168673 19077 raft_consensus.cc:729] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3afaa5ae6f4b0495783d77239efbd9, State: Initialized, Role: FOLLOWER
I20260926 14:39:34.168782 19077 consensus_queue.cc:260] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:34.169011 19077 ts_tablet_manager.cc:1428] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:34.331895 19060 raft_consensus.cc:482] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:34.332019 19060 raft_consensus.cc:504] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:34.332415 19060 leader_election.cc:288] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091), 1d3afaa5ae6f4b0495783d77239efbd9 (127.16.90.195:36907)
I20260926 14:39:34.332856 18634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60417ca00a1040388b5c8922845d13f9" candidate_uuid: "f576964c0138491e956d8198c2495d82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" is_pre_election: true
I20260926 14:39:34.332856 18915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60417ca00a1040388b5c8922845d13f9" candidate_uuid: "f576964c0138491e956d8198c2495d82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" is_pre_election: true
I20260926 14:39:34.333019 18634 raft_consensus.cc:2454] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f576964c0138491e956d8198c2495d82 in term 0.
I20260926 14:39:34.333019 18915 raft_consensus.cc:2454] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f576964c0138491e956d8198c2495d82 in term 0.
I20260926 14:39:34.333282 18706 leader_election.cc:302] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ebc3c6a60f4ee0af190a7b6fc230bc, f576964c0138491e956d8198c2495d82; no voters:
I20260926 14:39:34.333465 19060 raft_consensus.cc:2781] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:34.333527 19060 raft_consensus.cc:482] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:34.333552 19060 raft_consensus.cc:3037] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:34.334523 19060 raft_consensus.cc:504] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:34.334818 19060 leader_election.cc:288] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [CANDIDATE]: Term 1 election: Requested vote from peers 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091), 1d3afaa5ae6f4b0495783d77239efbd9 (127.16.90.195:36907)
I20260926 14:39:34.335132 18915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60417ca00a1040388b5c8922845d13f9" candidate_uuid: "f576964c0138491e956d8198c2495d82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3afaa5ae6f4b0495783d77239efbd9"
I20260926 14:39:34.335134 18634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60417ca00a1040388b5c8922845d13f9" candidate_uuid: "f576964c0138491e956d8198c2495d82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc"
I20260926 14:39:34.335258 18634 raft_consensus.cc:3037] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:34.335253 18915 raft_consensus.cc:3037] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:34.336356 18634 raft_consensus.cc:2454] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f576964c0138491e956d8198c2495d82 in term 1.
I20260926 14:39:34.336356 18915 raft_consensus.cc:2454] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f576964c0138491e956d8198c2495d82 in term 1.
I20260926 14:39:34.336615 18709 leader_election.cc:302] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3afaa5ae6f4b0495783d77239efbd9, f576964c0138491e956d8198c2495d82; no voters:
I20260926 14:39:34.336786 19060 raft_consensus.cc:2781] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:34.336956 19060 raft_consensus.cc:686] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 1 LEADER]: Becoming Leader. State: Replica: f576964c0138491e956d8198c2495d82, State: Running, Role: LEADER
I20260926 14:39:34.337090 19060 consensus_queue.cc:237] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } }
I20260926 14:39:34.338617 18476 catalog_manager.cc:5495] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 reported cstate change: term changed from 0 to 1, leader changed from <none> to f576964c0138491e956d8198c2495d82 (127.16.90.194). New cstate: current_term: 1 leader_uuid: "f576964c0138491e956d8198c2495d82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f576964c0138491e956d8198c2495d82" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 } health_report { overall_health: UNKNOWN } } }
I20260926 14:39:34.365275 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:34.362257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.16.90.194:37255)
0926 14:39:34.362496 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:34.362514 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:34.362525 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:39:34.362559 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:34.362561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:34.362599 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:34.362666 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:34.362921 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:34.363446 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:34.364296 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:34.364521 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:34.364604 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:34.364846 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:34.364918 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:34.364974 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:34.365065 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 14:39:34.365119 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260926 14:39:34.370564 18915 raft_consensus.cc:1261] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Refusing update from remote peer f576964c0138491e956d8198c2495d82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:34.370664 18634 raft_consensus.cc:1261] T 60417ca00a1040388b5c8922845d13f9 P 35ebc3c6a60f4ee0af190a7b6fc230bc [term 1 FOLLOWER]: Refusing update from remote peer f576964c0138491e956d8198c2495d82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:34.370935 19094 consensus_queue.cc:1035] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d3afaa5ae6f4b0495783d77239efbd9" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 36907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:34.371080 19060 consensus_queue.cc:1035] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ebc3c6a60f4ee0af190a7b6fc230bc" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:39:34.382468 19049 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 14:39:35.930521 18477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:35.930739 18477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:35.933332 18477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f044fcb40db14a989d6e65a01990dad8: Sending DeleteTablet for 3 replicas of tablet 60417ca00a1040388b5c8922845d13f9
I20260926 14:39:35.933924 18615 tablet_service.cc:1514] Processing DeleteTablet for tablet 60417ca00a1040388b5c8922845d13f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:35 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:36836
I20260926 14:39:35.933951 18755 tablet_service.cc:1514] Processing DeleteTablet for tablet 60417ca00a1040388b5c8922845d13f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:35 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:43252
I20260926 14:39:35.934206 18894 tablet_service.cc:1514] Processing DeleteTablet for tablet 60417ca00a1040388b5c8922845d13f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:35 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:40334
I20260926 14:39:35.934365 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 18550
I20260926 14:39:35.935343 19124 tablet_replica.cc:330] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82: stopping tablet replica
I20260926 14:39:35.935473 19124 raft_consensus.cc:2229] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:39:35.935647 19126 tablet_replica.cc:330] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9: stopping tablet replica
I20260926 14:39:35.935690 19124 raft_consensus.cc:2258] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:35.935847 19126 raft_consensus.cc:2229] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:35.935968 19126 raft_consensus.cc:2258] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:35.936332 19124 ts_tablet_manager.cc:1905] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:35.936661 19126 ts_tablet_manager.cc:1905] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:35.938716 19126 ts_tablet_manager.cc:1918] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:39:35.938787 19126 log.cc:1196] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/60417ca00a1040388b5c8922845d13f9
I20260926 14:39:35.938891 19124 ts_tablet_manager.cc:1918] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:39:35.938966 19124 log.cc:1196] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/60417ca00a1040388b5c8922845d13f9
I20260926 14:39:35.939033 19126 ts_tablet_manager.cc:1939] T 60417ca00a1040388b5c8922845d13f9 P 1d3afaa5ae6f4b0495783d77239efbd9: Deleting consensus metadata
I20260926 14:39:35.939236 19124 ts_tablet_manager.cc:1939] T 60417ca00a1040388b5c8922845d13f9 P f576964c0138491e956d8198c2495d82: Deleting consensus metadata
I20260926 14:39:35.939934 18465 catalog_manager.cc:4841] TS 1d3afaa5ae6f4b0495783d77239efbd9 (127.16.90.195:36907): tablet 60417ca00a1040388b5c8922845d13f9 (table test-table [id=9ec8b3703c98452081de76171c886566]) successfully deleted
I20260926 14:39:35.939992 18464 catalog_manager.cc:4841] TS f576964c0138491e956d8198c2495d82 (127.16.90.194:37255): tablet 60417ca00a1040388b5c8922845d13f9 (table test-table [id=9ec8b3703c98452081de76171c886566]) successfully deleted
W20260926 14:39:35.941069 18462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:34091 (error 108)
I20260926 14:39:35.941804 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 18690
I20260926 14:39:35.942108 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:35.941731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:34091 (local address 127.0.0.1:40694)
0926 14:39:35.941943 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:35.942014 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:34091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
W20260926 14:39:35.942443 18462 catalog_manager.cc:4584] TS 35ebc3c6a60f4ee0af190a7b6fc230bc (127.16.90.193:34091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60417ca00a1040388b5c8922845d13f9: Network error: Client connection negotiation failed: client connection to 127.16.90.193:34091: connect: Connection refused (error 111)
I20260926 14:39:35.948194 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 18830
I20260926 14:39:35.954032 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 18446
2026-09-26T14:39:35Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6689 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:39:36 dist-test-slave-x4sn krb5kdc[19131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:39:36 dist-test-slave-x4sn krb5kdc[19131](info): set up 2 sockets
Sep 26 14:39:36 dist-test-slave-x4sn krb5kdc[19131](info): commencing operation
krb5kdc: starting...
W20260926 14:39:36.905162 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:39:36 dist-test-slave-x4sn krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:39:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:39:37.018610 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:33691
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:33691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:37.125198 19144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:37.125512 19144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:37.125584 19144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:37.130370 19144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:37.130445 19144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:37.130471 19144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:37.130494 19144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:37.130520 19144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:37.137773 19144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:33691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:33691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:37.139156 19144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:37.140373 19144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:37.149070 19150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:37.149140 19149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:37.149109 19152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:37.149338 19144 server_base.cc:1034] running on GCE node
I20260926 14:39:37.149958 19144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:37.150990 19144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:37.152132 19144 hybrid_clock.cc:648] HybridClock initialized: now 1790433577152107 us; error 40 us; skew 500 ppm
Sep 26 14:39:37 dist-test-slave-x4sn krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:37.154749 19144 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:39:37.155565 19144 webserver.cc:460] Webserver started at http://127.16.90.254:35831/ using document root <none> and password file <none>
I20260926 14:39:37.156121 19144 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:37.156186 19144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:37.156399 19144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:37.158346 19144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "4bd00d168e424ad387dac806676ba633"
format_stamp: "Formatted at 2026-09-26 14:39:37 on dist-test-slave-x4sn"
server_key: "bf804a416757b798d7396e9b8aceafdd"
server_key_iv: "aeb7ae5b2b69cc139765b7a82d79323d"
server_key_version: "encryptionkey@0"
I20260926 14:39:37.158862 19144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "4bd00d168e424ad387dac806676ba633"
format_stamp: "Formatted at 2026-09-26 14:39:37 on dist-test-slave-x4sn"
server_key: "bf804a416757b798d7396e9b8aceafdd"
server_key_iv: "aeb7ae5b2b69cc139765b7a82d79323d"
server_key_version: "encryptionkey@0"
I20260926 14:39:37.162294 19144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:39:37.164562 19159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:37.165794 19144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:37.165899 19144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "4bd00d168e424ad387dac806676ba633"
format_stamp: "Formatted at 2026-09-26 14:39:37 on dist-test-slave-x4sn"
server_key: "bf804a416757b798d7396e9b8aceafdd"
server_key_iv: "aeb7ae5b2b69cc139765b7a82d79323d"
server_key_version: "encryptionkey@0"
I20260926 14:39:37.165989 19144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:37.196198 19144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:37.198886 19144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:37.199056 19144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:37.205888 19144 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:33691
I20260926 14:39:37.205912 19211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:33691 every 8 connection(s)
I20260926 14:39:37.206804 19144 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:37.209475 19212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:37.214358 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 19144
I20260926 14:39:37.214473 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:37.214740 16747 external_mini_cluster.cc:1428] Setting key 95aa606b4d7d9db2fd1344b1a0e485f7
I20260926 14:39:37.215034 19212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633: Bootstrap starting.
I20260926 14:39:37.217440 19212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:37.218150 19212 log.cc:824] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:37.219982 19212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633: No bootstrap required, opened a new log
Sep 26 14:39:37 dist-test-slave-x4sn krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:37.223315 19212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd00d168e424ad387dac806676ba633" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33691 } }
I20260926 14:39:37.223536 19212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:37.223592 19212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd00d168e424ad387dac806676ba633, State: Initialized, Role: FOLLOWER
I20260926 14:39:37.224035 19212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd00d168e424ad387dac806676ba633" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33691 } }
I20260926 14:39:37.224193 19212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:37.224264 19212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:37.224371 19212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:37.225147 19212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd00d168e424ad387dac806676ba633" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33691 } }
I20260926 14:39:37.225450 19212 leader_election.cc:302] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd00d168e424ad387dac806676ba633; no voters:
I20260926 14:39:37.225744 19212 leader_election.cc:288] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:37.225839 19217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:37.226040 19217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [term 1 LEADER]: Becoming Leader. State: Replica: 4bd00d168e424ad387dac806676ba633, State: Running, Role: LEADER
I20260926 14:39:37.226320 19217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd00d168e424ad387dac806676ba633" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33691 } }
I20260926 14:39:37.227002 19212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:37.227885 19219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bd00d168e424ad387dac806676ba633. Latest consensus state: current_term: 1 leader_uuid: "4bd00d168e424ad387dac806676ba633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd00d168e424ad387dac806676ba633" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33691 } } }
I20260926 14:39:37.228001 19219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:37.228020 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:37.216034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.16.90.254:33691)
0926 14:39:37.216416 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:37.216431 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:37.216465 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:39:37.217073 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:37.217090 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:37.217415 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:37.217737 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:37.218542 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.219488 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.220395 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.220851 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.221005 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:37.223212 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:37.223216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:37.223228 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:37.223253 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:37.224993 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:37.225394 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:37.225400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:37.225405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:37.225511 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:37.225775 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:37.225779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:37.225780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:37.226280 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:37.226474 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:37.226725 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 14:39:37.227016 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
I20260926 14:39:37.228273 19218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bd00d168e424ad387dac806676ba633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd00d168e424ad387dac806676ba633" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33691 } } }
I20260926 14:39:37.228358 19218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633 [sys.catalog]: This master's current role is: LEADER
W20260926 14:39:37.232059 19232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:39:37.232142 19232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:39:37.232206 19226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:37.232841 19226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:37.237918 19226 catalog_manager.cc:1347] Generated new cluster ID: 036eaa238a154b5e808fdb4b01ae8348
I20260926 14:39:37.237982 19226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:37.271100 19226 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:37.272432 19226 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:37.278119 19226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bd00d168e424ad387dac806676ba633: Generated new TSK 0
I20260926 14:39:37.278898 19226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:39:37.304795 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:33691
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:37.410275 19238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:37.410521 19238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:37.410560 19238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:37.414726 19238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:37.414808 19238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:37.414870 19238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:37.420307 19238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:37.421286 19238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:37.422256 19238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:37.431447 19246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:37.431456 19243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:37.431526 19244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:37.431716 19238 server_base.cc:1034] running on GCE node
I20260926 14:39:37.432202 19238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:37.432782 19238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:37.433993 19238 hybrid_clock.cc:648] HybridClock initialized: now 1790433577433966 us; error 39 us; skew 500 ppm
Sep 26 14:39:37 dist-test-slave-x4sn krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:37.436563 19238 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:39:37.437338 19238 webserver.cc:460] Webserver started at http://127.16.90.193:38213/ using document root <none> and password file <none>
I20260926 14:39:37.437824 19238 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:37.437876 19238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:37.438030 19238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:37.439988 19238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "ca1af21ee093471090c4423aadd272b1"
format_stamp: "Formatted at 2026-09-26 14:39:37 on dist-test-slave-x4sn"
server_key: "f40a2dbc2bad73e92cf73b4b279070ee"
server_key_iv: "864f3037e771a52e76f36a092f5dc900"
server_key_version: "encryptionkey@0"
I20260926 14:39:37.440488 19238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "ca1af21ee093471090c4423aadd272b1"
format_stamp: "Formatted at 2026-09-26 14:39:37 on dist-test-slave-x4sn"
server_key: "f40a2dbc2bad73e92cf73b4b279070ee"
server_key_iv: "864f3037e771a52e76f36a092f5dc900"
server_key_version: "encryptionkey@0"
I20260926 14:39:37.444025 19238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:37.446429 19253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:37.447860 19238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:39:37.448016 19238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "ca1af21ee093471090c4423aadd272b1"
format_stamp: "Formatted at 2026-09-26 14:39:37 on dist-test-slave-x4sn"
server_key: "f40a2dbc2bad73e92cf73b4b279070ee"
server_key_iv: "864f3037e771a52e76f36a092f5dc900"
server_key_version: "encryptionkey@0"
I20260926 14:39:37.448125 19238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:37.496699 19238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:37.499706 19238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:37.499902 19238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:37.500522 19238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:37.501390 19238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:37.501441 19238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:37.501500 19238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:37.501539 19238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:37.510680 19238 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:37111
I20260926 14:39:37.510721 19366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:37111 every 8 connection(s)
I20260926 14:39:37.511598 19238 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:39:37 dist-test-slave-x4sn krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:37.521142 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 19238
I20260926 14:39:37.521243 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:39:37.521539 16747 external_mini_cluster.cc:1428] Setting key de200796018759c306dd11610dba5ac4
I20260926 14:39:37.524339 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:37.513404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:59013 (local address 127.16.90.254:33691)
0926 14:39:37.513546 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:37.513551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:37.514382 (+ 831us) server_negotiation.cc:407] Connection header received
0926 14:39:37.515576 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:37.515579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:37.515623 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:37.515715 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:37.516809 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.517326 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.518380 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.518611 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.518694 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:37.520900 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:37.520904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:37.520907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:37.520935 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:37.522346 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:37.522803 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:37.522807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:37.522809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:37.522898 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:37.523292 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:37.523295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:37.523296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:37.523460 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:37.523532 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:37.524101 (+ 569us) server_negotiation.cc:299] Negotiation successful
0926 14:39:37.524223 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:39:37.525239 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:37.513781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:33691 (local address 127.16.90.193:59013)
0926 14:39:37.514230 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:37.514272 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:37.515396 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:37.515881 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:37.515887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:37.516226 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:37.516690 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:37.516703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.517412 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.517414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:37.518269 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:37.518277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.518734 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.518737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:37.518865 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:37.519343 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:37.519359 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:37.520752 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:37.522448 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:37.522453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:37.522463 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:37.522705 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:37.522998 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:37.523001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:37.523003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:37.523202 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:37.523551 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:37.523558 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:37.523879 (+ 321us) client_negotiation.cc:769] Sending connection context
0926 14:39:37.524073 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 14:39:37.524392 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
I20260926 14:39:37.526342 19367 heartbeater.cc:347] Connected to a master server at 127.16.90.254:33691
I20260926 14:39:37.526575 19367 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:37.527149 19367 heartbeater.cc:511] Master 127.16.90.254:33691 requested a full tablet report, sending...
I20260926 14:39:37.528719 19176 ts_manager.cc:194] Registered new tserver with Master: ca1af21ee093471090c4423aadd272b1 (127.16.90.193:37111)
I20260926 14:39:37.529814 19176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:59013
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:39:37.547760 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:33691
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:37.654554 19372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:37.654831 19372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:37.654872 19372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:37.658885 19372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:37.658941 19372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:37.658998 19372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:37.664067 19372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:37.665053 19372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:37.666030 19372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:37.675164 19377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:37.675233 19378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:37.675257 19380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:37.675360 19372 server_base.cc:1034] running on GCE node
I20260926 14:39:37.675856 19372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:37.676445 19372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:37.677598 19372 hybrid_clock.cc:648] HybridClock initialized: now 1790433577677575 us; error 34 us; skew 500 ppm
Sep 26 14:39:37 dist-test-slave-x4sn krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:37.680150 19372 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:39:37.680972 19372 webserver.cc:460] Webserver started at http://127.16.90.194:33115/ using document root <none> and password file <none>
I20260926 14:39:37.681465 19372 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:37.681520 19372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:37.681685 19372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:37.683637 19372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "38dc77fd615b465484a28d66f701da4d"
format_stamp: "Formatted at 2026-09-26 14:39:37 on dist-test-slave-x4sn"
server_key: "396157c3bfc8f43433a7c9763d5b9bb2"
server_key_iv: "32201364c105a0695a200d6af5732600"
server_key_version: "encryptionkey@0"
I20260926 14:39:37.684268 19372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "38dc77fd615b465484a28d66f701da4d"
format_stamp: "Formatted at 2026-09-26 14:39:37 on dist-test-slave-x4sn"
server_key: "396157c3bfc8f43433a7c9763d5b9bb2"
server_key_iv: "32201364c105a0695a200d6af5732600"
server_key_version: "encryptionkey@0"
I20260926 14:39:37.687867 19372 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:39:37.690248 19387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:37.691404 19372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:37.691501 19372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "38dc77fd615b465484a28d66f701da4d"
format_stamp: "Formatted at 2026-09-26 14:39:37 on dist-test-slave-x4sn"
server_key: "396157c3bfc8f43433a7c9763d5b9bb2"
server_key_iv: "32201364c105a0695a200d6af5732600"
server_key_version: "encryptionkey@0"
I20260926 14:39:37.691605 19372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:37.736773 19372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:37.739611 19372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:37.739820 19372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:37.740414 19372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:37.741292 19372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:37.741338 19372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:37.741381 19372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:37.741408 19372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:37.750085 19372 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:39719
I20260926 14:39:37.750106 19500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:39719 every 8 connection(s)
I20260926 14:39:37.751036 19372 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:39:37.753526 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 19372
I20260926 14:39:37.753628 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:39:37.753887 16747 external_mini_cluster.cc:1428] Setting key 134b7de995e2de1e198de35c1771b198
Sep 26 14:39:37 dist-test-slave-x4sn krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
I20260926 14:39:37.764988 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:37.752943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:37271 (local address 127.16.90.254:33691)
0926 14:39:37.753091 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:37.753096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:37.754002 (+ 906us) server_negotiation.cc:407] Connection header received
0926 14:39:37.755481 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:37.755485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:37.755534 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:37.755609 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:37.756799 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.757423 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.758534 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.758755 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.758842 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:37.761227 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:37.761231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:37.761233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:37.761258 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:37.762913 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:37.763382 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:37.763385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:37.763387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:37.763467 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:37.763851 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:37.763854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:37.763856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:37.764060 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:37.764145 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:37.764651 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 14:39:37.764810 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
I20260926 14:39:37.765928 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:37.753380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:33691 (local address 127.16.90.194:37271)
0926 14:39:37.753845 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:37.753899 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:37.755282 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:37.755819 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:37.755827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:37.756177 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:37.756664 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:37.756677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.757536 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.757539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:37.758419 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:37.758428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.758892 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.758896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:37.759065 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:37.759602 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:37.759619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:37.761044 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:37.763023 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:37.763029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:37.763045 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:37.763290 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:37.763559 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:37.763562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:37.763564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:37.763764 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:37.764155 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:37.764163 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:37.764416 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 14:39:37.764597 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 14:39:37.764935 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
I20260926 14:39:37.767024 19501 heartbeater.cc:347] Connected to a master server at 127.16.90.254:33691
I20260926 14:39:37.767303 19501 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:37.767822 19501 heartbeater.cc:511] Master 127.16.90.254:33691 requested a full tablet report, sending...
I20260926 14:39:37.768839 19176 ts_manager.cc:194] Registered new tserver with Master: 38dc77fd615b465484a28d66f701da4d (127.16.90.194:39719)
I20260926 14:39:37.769634 19176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:37271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:39:37.782256 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:33691
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:37.888804 19506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:37.889041 19506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:37.889081 19506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:37.892953 19506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:37.893008 19506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:37.893062 19506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:37.898149 19506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:37.899080 19506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:37.900059 19506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:37.909109 19514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:37.909118 19511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:37.909216 19512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:37.909397 19506 server_base.cc:1034] running on GCE node
I20260926 14:39:37.909786 19506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:37.910369 19506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:37.911527 19506 hybrid_clock.cc:648] HybridClock initialized: now 1790433577911501 us; error 37 us; skew 500 ppm
Sep 26 14:39:37 dist-test-slave-x4sn krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:37.914132 19506 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:39:37.914969 19506 webserver.cc:460] Webserver started at http://127.16.90.195:32873/ using document root <none> and password file <none>
I20260926 14:39:37.915484 19506 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:37.915547 19506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:37.915774 19506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:37.917763 19506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "e71493b5a1814a53acf85ba187c5151e"
format_stamp: "Formatted at 2026-09-26 14:39:37 on dist-test-slave-x4sn"
server_key: "c77aa23cda0220aac8c5db72bc98f68b"
server_key_iv: "893a5a09b556d4fb0e6e6c0c9686bddc"
server_key_version: "encryptionkey@0"
I20260926 14:39:37.918279 19506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "e71493b5a1814a53acf85ba187c5151e"
format_stamp: "Formatted at 2026-09-26 14:39:37 on dist-test-slave-x4sn"
server_key: "c77aa23cda0220aac8c5db72bc98f68b"
server_key_iv: "893a5a09b556d4fb0e6e6c0c9686bddc"
server_key_version: "encryptionkey@0"
I20260926 14:39:37.921761 19506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:39:37.923992 19521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:37.925125 19506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:37.925238 19506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "e71493b5a1814a53acf85ba187c5151e"
format_stamp: "Formatted at 2026-09-26 14:39:37 on dist-test-slave-x4sn"
server_key: "c77aa23cda0220aac8c5db72bc98f68b"
server_key_iv: "893a5a09b556d4fb0e6e6c0c9686bddc"
server_key_version: "encryptionkey@0"
I20260926 14:39:37.925323 19506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:37.936069 19506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:37.938987 19506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:37.939164 19506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:37.939726 19506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:37.940505 19506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:37.940551 19506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:37.940603 19506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:37.940631 19506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:37.948652 19506 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:39237
I20260926 14:39:37.948671 19634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:39237 every 8 connection(s)
I20260926 14:39:37.949486 19506 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:37.958046 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 19506
I20260926 14:39:37.958173 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:39:37.958527 16747 external_mini_cluster.cc:1428] Setting key ed508816f0280a80e2eff15896b2dca1
Sep 26 14:39:37 dist-test-slave-x4sn krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:37.962356 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:37.951159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:55247 (local address 127.16.90.254:33691)
0926 14:39:37.951295 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:37.951298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:37.952132 (+ 834us) server_negotiation.cc:407] Connection header received
0926 14:39:37.953430 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:37.953434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:37.953489 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:37.953563 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:37.954697 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.955227 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.956392 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.956607 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.956694 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:37.959155 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:37.959159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:37.959162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:37.959199 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:37.960626 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:37.961050 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:37.961052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:37.961053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:37.961117 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:37.961487 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:37.961489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:37.961490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:37.961650 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:37.961734 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:37.962138 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 14:39:37.962241 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:39:37.963243 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:37.951527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:33691 (local address 127.16.90.195:55247)
0926 14:39:37.951997 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:37.952038 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:37.953229 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:37.953750 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:37.953757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:37.954092 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:37.954559 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:37.954571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.955342 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.955345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:37.956277 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:37.956287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:37.956745 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:37.956749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:37.956901 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:37.957375 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:37.957391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:37.958971 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:37.960714 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:37.960720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:37.960735 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:37.960976 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:37.961265 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:37.961268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:37.961269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:37.961411 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:37.961734 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:37.961738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:37.961960 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 14:39:37.962111 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 14:39:37.962370 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":153,"threads_started":1}
I20260926 14:39:37.964360 19635 heartbeater.cc:347] Connected to a master server at 127.16.90.254:33691
I20260926 14:39:37.964596 19635 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:37.965062 19635 heartbeater.cc:511] Master 127.16.90.254:33691 requested a full tablet report, sending...
I20260926 14:39:37.966023 19176 ts_manager.cc:194] Registered new tserver with Master: e71493b5a1814a53acf85ba187c5151e (127.16.90.195:39237)
I20260926 14:39:37.966553 19176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:55247
I20260926 14:39:37.972249 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 14:39:37.984277 19215 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:39:37.982886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.16.90.254:33691)
0926 14:39:37.983038 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:37.983043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:37.983156 (+ 113us) server_negotiation.cc:407] Connection header received
0926 14:39:37.983264 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:37.983266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:37.983308 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:37.983375 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:37.984139 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54130: BlockingRecv error: recv got EOF from 127.0.0.1:54130 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 14:39:37.985435 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 19238
I20260926 14:39:37.990206 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 19372
I20260926 14:39:37.994402 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 19506
I20260926 14:39:37.998924 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 19144
2026-09-26T14:39:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2043 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:39:38 dist-test-slave-x4sn krb5kdc[19647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:39:38 dist-test-slave-x4sn krb5kdc[19647](info): set up 2 sockets
Sep 26 14:39:38 dist-test-slave-x4sn krb5kdc[19647](info): commencing operation
krb5kdc: starting...
W20260926 14:39:38.944974 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:39:39 dist-test-slave-x4sn krb5kdc[19647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:39:39.060825 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35023
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:35023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:39.168583 19660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:39.168870 19660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:39.168927 19660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:39.172953 19660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:39.173015 19660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:39.173043 19660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:39.173064 19660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:39.173092 19660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:39.178337 19660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:35023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:39.179402 19660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:39.180452 19660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:39.188469 19665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:39.188506 19666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:39.188477 19668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:39.188740 19660 server_base.cc:1034] running on GCE node
I20260926 14:39:39.189318 19660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:39.190384 19660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:39.191542 19660 hybrid_clock.cc:648] HybridClock initialized: now 1790433579191522 us; error 38 us; skew 500 ppm
Sep 26 14:39:39 dist-test-slave-x4sn krb5kdc[19647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433579, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:39.206434 19660 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:39:39.207481 19660 webserver.cc:460] Webserver started at http://127.16.90.254:46159/ using document root <none> and password file <none>
I20260926 14:39:39.208101 19660 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:39.208171 19660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:39.208369 19660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:39.210441 19660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "406981557dc54ec68ee11da4723d64e0"
format_stamp: "Formatted at 2026-09-26 14:39:39 on dist-test-slave-x4sn"
server_key: "1413d0cf35f3737e712faec2e2978ccc"
server_key_iv: "1776474d96d34dafd7e0c311b21aae4f"
server_key_version: "encryptionkey@0"
I20260926 14:39:39.210978 19660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "406981557dc54ec68ee11da4723d64e0"
format_stamp: "Formatted at 2026-09-26 14:39:39 on dist-test-slave-x4sn"
server_key: "1413d0cf35f3737e712faec2e2978ccc"
server_key_iv: "1776474d96d34dafd7e0c311b21aae4f"
server_key_version: "encryptionkey@0"
I20260926 14:39:39.214669 19660 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:39:39.217186 19675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:39.218443 19660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:39:39.218547 19660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "406981557dc54ec68ee11da4723d64e0"
format_stamp: "Formatted at 2026-09-26 14:39:39 on dist-test-slave-x4sn"
server_key: "1413d0cf35f3737e712faec2e2978ccc"
server_key_iv: "1776474d96d34dafd7e0c311b21aae4f"
server_key_version: "encryptionkey@0"
I20260926 14:39:39.218648 19660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:39.229321 19660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:39.232359 19660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:39.232534 19660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:39.239319 19660 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:35023
I20260926 14:39:39.239619 19727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:35023 every 8 connection(s)
I20260926 14:39:39.240284 19660 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:39.243069 19728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:39.246795 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 19660
I20260926 14:39:39.246893 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:39.247151 16747 external_mini_cluster.cc:1428] Setting key 3e39fae51fd959545b0584e8c8bda6e6
I20260926 14:39:39.248700 19728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0: Bootstrap starting.
I20260926 14:39:39.251029 19728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:39.251780 19728 log.cc:824] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:39.253877 19728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0: No bootstrap required, opened a new log
Sep 26 14:39:39 dist-test-slave-x4sn krb5kdc[19647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:39.257881 19728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "406981557dc54ec68ee11da4723d64e0" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35023 } }
I20260926 14:39:39.258108 19728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:39.258143 19728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 406981557dc54ec68ee11da4723d64e0, State: Initialized, Role: FOLLOWER
I20260926 14:39:39.258594 19728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "406981557dc54ec68ee11da4723d64e0" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35023 } }
I20260926 14:39:39.258715 19728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:39.258776 19728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:39.258863 19728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:39.259771 19728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "406981557dc54ec68ee11da4723d64e0" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35023 } }
I20260926 14:39:39.260082 19728 leader_election.cc:302] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 406981557dc54ec68ee11da4723d64e0; no voters:
I20260926 14:39:39.260380 19728 leader_election.cc:288] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:39.260509 19733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:39.260717 19733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [term 1 LEADER]: Becoming Leader. State: Replica: 406981557dc54ec68ee11da4723d64e0, State: Running, Role: LEADER
I20260926 14:39:39.261097 19733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "406981557dc54ec68ee11da4723d64e0" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35023 } }
I20260926 14:39:39.261612 19728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:39.261695 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:39.248432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.16.90.254:35023)
0926 14:39:39.248841 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:39.248861 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:39.248897 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:39:39.249528 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:39.249544 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:39.249880 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:39.250154 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:39.251021 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:39.252130 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:39.253339 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:39.253817 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:39.253937 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:39.256595 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:39.256601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:39.256616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:39.256643 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:39.258273 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:39.258687 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:39.258694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:39.258700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:39.258849 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:39.259119 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:39.259122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:39.259124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:39.259553 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:39.259767 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:39.260350 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 14:39:39.260682 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260926 14:39:39.262355 19733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 406981557dc54ec68ee11da4723d64e0. Latest consensus state: current_term: 1 leader_uuid: "406981557dc54ec68ee11da4723d64e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "406981557dc54ec68ee11da4723d64e0" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35023 } } }
I20260926 14:39:39.263026 19733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:39.262920 19734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "406981557dc54ec68ee11da4723d64e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "406981557dc54ec68ee11da4723d64e0" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35023 } } }
I20260926 14:39:39.263267 19734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:39.263536 19743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:39.264892 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:39:39.267174 19743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:39.272552 19743 catalog_manager.cc:1347] Generated new cluster ID: e8279646de7e40abb4ce4d241db00432
I20260926 14:39:39.272609 19743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:39.283782 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:39.278250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.16.90.254:35023)
0926 14:39:39.278356 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:39.278361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:39.278630 (+ 269us) server_negotiation.cc:407] Connection header received
0926 14:39:39.278741 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:39.278744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:39.278797 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:39.278878 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:39.279403 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:39.280278 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:39.281380 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:39.281767 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:39.281859 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:39.282728 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:39.282732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:39.282735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:39.283293 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:39.283327 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:39.283347 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:39.283430 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:39.283515 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 14:39:39.283578 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:39:39.290237 19743 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:39.291455 19743 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:39.298689 19743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 406981557dc54ec68ee11da4723d64e0: Generated new TSK 0
I20260926 14:39:39.299474 19743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:39:39.306576 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 19660
2026-09-26T14:39:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1306 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:39:39 dist-test-slave-x4sn krb5kdc[19761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:39:39 dist-test-slave-x4sn krb5kdc[19761](info): set up 2 sockets
Sep 26 14:39:39 dist-test-slave-x4sn krb5kdc[19761](info): commencing operation
krb5kdc: starting...
W20260926 14:39:40.253153 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:39:40 dist-test-slave-x4sn krb5kdc[19761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:39:40.366539 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:43913
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:43913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:40.472618 19774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:40.472898 19774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:40.472958 19774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:40.476924 19774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:40.476982 19774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:40.477008 19774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:40.477029 19774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:40.477058 19774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:40.482223 19774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:43913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:43913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:40.483261 19774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:40.484313 19774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:40.492422 19780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:40.492426 19779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:40.492445 19782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:40.492558 19774 server_base.cc:1034] running on GCE node
I20260926 14:39:40.493257 19774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:40.494366 19774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:40.495518 19774 hybrid_clock.cc:648] HybridClock initialized: now 1790433580495504 us; error 29 us; skew 500 ppm
Sep 26 14:39:40 dist-test-slave-x4sn krb5kdc[19761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:40.498070 19774 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:39:40.498905 19774 webserver.cc:460] Webserver started at http://127.16.90.254:39621/ using document root <none> and password file <none>
I20260926 14:39:40.499423 19774 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:40.499486 19774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:40.499733 19774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:40.501778 19774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "5288da2e2f1d4a0eb18025eb1779008a"
format_stamp: "Formatted at 2026-09-26 14:39:40 on dist-test-slave-x4sn"
server_key: "c3af8bbeecb902fdd8495ff343aa25fe"
server_key_iv: "3a8016a42a8fb49f35471fe65e92078b"
server_key_version: "encryptionkey@0"
I20260926 14:39:40.502307 19774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "5288da2e2f1d4a0eb18025eb1779008a"
format_stamp: "Formatted at 2026-09-26 14:39:40 on dist-test-slave-x4sn"
server_key: "c3af8bbeecb902fdd8495ff343aa25fe"
server_key_iv: "3a8016a42a8fb49f35471fe65e92078b"
server_key_version: "encryptionkey@0"
I20260926 14:39:40.505891 19774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 14:39:40.508194 19789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:40.509344 19774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:40.509451 19774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "5288da2e2f1d4a0eb18025eb1779008a"
format_stamp: "Formatted at 2026-09-26 14:39:40 on dist-test-slave-x4sn"
server_key: "c3af8bbeecb902fdd8495ff343aa25fe"
server_key_iv: "3a8016a42a8fb49f35471fe65e92078b"
server_key_version: "encryptionkey@0"
I20260926 14:39:40.509538 19774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:40.526751 19774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:40.529536 19774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:40.529716 19774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:40.536449 19774 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:43913
I20260926 14:39:40.536721 19841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:43913 every 8 connection(s)
I20260926 14:39:40.537401 19774 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:40.540189 19842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:40.541921 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 19774
I20260926 14:39:40.542035 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:40.542294 16747 external_mini_cluster.cc:1428] Setting key e985a194c69328d7f26375d969800fd4
I20260926 14:39:40.546955 19842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a: Bootstrap starting.
I20260926 14:39:40.549606 19842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:40.550599 19842 log.cc:824] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:39:40 dist-test-slave-x4sn krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:40.552843 19842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a: No bootstrap required, opened a new log
I20260926 14:39:40.556778 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:40.543768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.16.90.254:43913)
0926 14:39:40.544208 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:40.544228 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:40.544271 (+ 43us) server_negotiation.cc:407] Connection header received
0926 14:39:40.545030 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:40.545055 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:40.545477 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:40.545873 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:40.546534 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:40.547990 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:40.548893 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:40.549567 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:40.549742 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:40.552327 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:40.552332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:40.552347 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:40.552373 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:40.553995 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:40.554475 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:40.554480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:40.554484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:40.554594 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:40.554857 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:40.554859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:40.554860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:40.555240 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:40.555380 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:40.555658 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 14:39:40.555998 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":126,"threads_started":1}
I20260926 14:39:40.556888 19842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5288da2e2f1d4a0eb18025eb1779008a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43913 } }
I20260926 14:39:40.557142 19842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:40.557197 19842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5288da2e2f1d4a0eb18025eb1779008a, State: Initialized, Role: FOLLOWER
I20260926 14:39:40.557595 19842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5288da2e2f1d4a0eb18025eb1779008a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43913 } }
I20260926 14:39:40.557729 19842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:40.557793 19842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:40.557893 19842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:40.558813 19842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5288da2e2f1d4a0eb18025eb1779008a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43913 } }
I20260926 14:39:40.559134 19842 leader_election.cc:302] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5288da2e2f1d4a0eb18025eb1779008a; no voters:
I20260926 14:39:40.559418 19842 leader_election.cc:288] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:40.559514 19847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:40.559762 19847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [term 1 LEADER]: Becoming Leader. State: Replica: 5288da2e2f1d4a0eb18025eb1779008a, State: Running, Role: LEADER
I20260926 14:39:40.560081 19847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5288da2e2f1d4a0eb18025eb1779008a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43913 } }
I20260926 14:39:40.560559 19842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:40.561576 19849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5288da2e2f1d4a0eb18025eb1779008a. Latest consensus state: current_term: 1 leader_uuid: "5288da2e2f1d4a0eb18025eb1779008a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5288da2e2f1d4a0eb18025eb1779008a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43913 } } }
I20260926 14:39:40.561771 19849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:40.562125 19848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5288da2e2f1d4a0eb18025eb1779008a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5288da2e2f1d4a0eb18025eb1779008a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43913 } } }
I20260926 14:39:40.562363 19848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a [sys.catalog]: This master's current role is: LEADER
W20260926 14:39:40.565105 19862 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:39:40.565166 19862 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:39:40.565238 19856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:40.565879 19856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:40.571053 19856 catalog_manager.cc:1347] Generated new cluster ID: 953b0d40362b41ffa724c9f0769fe36c
I20260926 14:39:40.571123 19856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:40.595198 19856 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:40.596257 19856 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:40.604597 19856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5288da2e2f1d4a0eb18025eb1779008a: Generated new TSK 0
I20260926 14:39:40.605304 19856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:39:40.636631 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43913
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:40.744966 19868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:40.745235 19868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:40.745276 19868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:40.749223 19868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:40.749279 19868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:40.749334 19868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:40.754482 19868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:40.755497 19868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:40.756561 19868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:40.765969 19873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:40.766013 19874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:40.766103 19876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:40.766018 19868 server_base.cc:1034] running on GCE node
I20260926 14:39:40.766497 19868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:40.767095 19868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:40.768242 19868 hybrid_clock.cc:648] HybridClock initialized: now 1790433580768219 us; error 41 us; skew 500 ppm
Sep 26 14:39:40 dist-test-slave-x4sn krb5kdc[19761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:40.770776 19868 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:39:40.771549 19868 webserver.cc:460] Webserver started at http://127.16.90.193:41371/ using document root <none> and password file <none>
I20260926 14:39:40.772104 19868 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:40.772169 19868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:40.772358 19868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:40.774296 19868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "5fe24cf5327e42e6bd718aa146c044a6"
format_stamp: "Formatted at 2026-09-26 14:39:40 on dist-test-slave-x4sn"
server_key: "ce7d3f19aab5d8aa45d3e127d4d88565"
server_key_iv: "ac99b7cbdd51ed357802165b9f809db2"
server_key_version: "encryptionkey@0"
I20260926 14:39:40.774814 19868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "5fe24cf5327e42e6bd718aa146c044a6"
format_stamp: "Formatted at 2026-09-26 14:39:40 on dist-test-slave-x4sn"
server_key: "ce7d3f19aab5d8aa45d3e127d4d88565"
server_key_iv: "ac99b7cbdd51ed357802165b9f809db2"
server_key_version: "encryptionkey@0"
I20260926 14:39:40.778216 19868 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:39:40.780457 19883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:40.781824 19868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:40.781924 19868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "5fe24cf5327e42e6bd718aa146c044a6"
format_stamp: "Formatted at 2026-09-26 14:39:40 on dist-test-slave-x4sn"
server_key: "ce7d3f19aab5d8aa45d3e127d4d88565"
server_key_iv: "ac99b7cbdd51ed357802165b9f809db2"
server_key_version: "encryptionkey@0"
I20260926 14:39:40.782012 19868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:40.800505 19868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:40.803498 19868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:40.803715 19868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:40.804297 19868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:40.805117 19868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:40.805166 19868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:40.805218 19868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:40.805245 19868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:40.813692 19868 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:45727
I20260926 14:39:40.813735 19996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:45727 every 8 connection(s)
I20260926 14:39:40.814559 19868 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:39:40.822274 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 19868
I20260926 14:39:40.822396 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:39:40.822669 16747 external_mini_cluster.cc:1428] Setting key e4571533809ff2806ff9cb0dfef2af4f
Sep 26 14:39:40 dist-test-slave-x4sn krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:40.828423 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:40.816263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:35573 (local address 127.16.90.254:43913)
0926 14:39:40.816399 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:40.816403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:40.817309 (+ 906us) server_negotiation.cc:407] Connection header received
0926 14:39:40.818812 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:40.818816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:40.818877 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:40.818962 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:40.820230 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:40.820788 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:40.821981 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:40.822211 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:40.822310 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:40.824945 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:40.824948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:40.824950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:40.824983 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:40.826356 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:40.826945 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:40.826947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:40.826948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:40.827017 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:40.827355 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:40.827357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:40.827359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:40.827509 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:40.827576 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:40.828204 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 14:39:40.828304 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:39:40.829594 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:40.816636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:43913 (local address 127.16.90.193:35573)
0926 14:39:40.817128 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:40.817191 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:40.818579 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:40.819142 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:40.819148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:40.819534 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:40.820074 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:40.820089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:40.820920 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:40.820924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:40.821841 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:40.821851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:40.822357 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:40.822362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:40.822555 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:40.823160 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:40.823178 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:40.824781 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:40.826478 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:40.826484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:40.826500 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:40.826852 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:40.827090 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:40.827093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:40.827095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:40.827273 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:40.827599 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:40.827606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:40.827943 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 14:39:40.828173 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 14:39:40.828530 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":113,"threads_started":1}
I20260926 14:39:40.830786 19997 heartbeater.cc:347] Connected to a master server at 127.16.90.254:43913
I20260926 14:39:40.830993 19997 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:40.831422 19997 heartbeater.cc:511] Master 127.16.90.254:43913 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
I20260926 14:39:40.832957 19806 ts_manager.cc:194] Registered new tserver with Master: 5fe24cf5327e42e6bd718aa146c044a6 (127.16.90.193:45727)
I20260926 14:39:40.834170 19806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:35573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:39:40.848562 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43913
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:40.953601 20002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:40.953862 20002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:40.953919 20002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:40.957818 20002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:40.957871 20002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:40.957947 20002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:40.963101 20002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:40.964143 20002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:40.965096 20002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:40.974031 20010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:40.974013 20008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:40.974016 20007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:40.974347 20002 server_base.cc:1034] running on GCE node
I20260926 14:39:40.974669 20002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:40.975255 20002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:40.976397 20002 hybrid_clock.cc:648] HybridClock initialized: now 1790433580976379 us; error 30 us; skew 500 ppm
Sep 26 14:39:40 dist-test-slave-x4sn krb5kdc[19761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:40.978813 20002 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:39:40.979609 20002 webserver.cc:460] Webserver started at http://127.16.90.194:38433/ using document root <none> and password file <none>
I20260926 14:39:40.980129 20002 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:40.980198 20002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:40.980394 20002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:40.982352 20002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "4412aa84d53548c79a889e6fb3d440da"
format_stamp: "Formatted at 2026-09-26 14:39:40 on dist-test-slave-x4sn"
server_key: "b6995b0560dbf2fe1b99ae16d232fc11"
server_key_iv: "3fcf00405b11b3e6a2aa10fd2044a1e8"
server_key_version: "encryptionkey@0"
I20260926 14:39:40.982858 20002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "4412aa84d53548c79a889e6fb3d440da"
format_stamp: "Formatted at 2026-09-26 14:39:40 on dist-test-slave-x4sn"
server_key: "b6995b0560dbf2fe1b99ae16d232fc11"
server_key_iv: "3fcf00405b11b3e6a2aa10fd2044a1e8"
server_key_version: "encryptionkey@0"
I20260926 14:39:40.986433 20002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:39:40.988684 20017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:40.989887 20002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:39:40.989995 20002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "4412aa84d53548c79a889e6fb3d440da"
format_stamp: "Formatted at 2026-09-26 14:39:40 on dist-test-slave-x4sn"
server_key: "b6995b0560dbf2fe1b99ae16d232fc11"
server_key_iv: "3fcf00405b11b3e6a2aa10fd2044a1e8"
server_key_version: "encryptionkey@0"
I20260926 14:39:40.990093 20002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:41.010668 20002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:41.013701 20002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:41.013880 20002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:41.014451 20002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:41.015239 20002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:41.015286 20002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:41.015336 20002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:41.015365 20002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:41.023464 20002 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:41201
I20260926 14:39:41.023509 20130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:41201 every 8 connection(s)
I20260926 14:39:41.024341 20002 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:39:41 dist-test-slave-x4sn krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:41.033946 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 20002
I20260926 14:39:41.034047 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:39:41.034320 16747 external_mini_cluster.cc:1428] Setting key 9cb3712f4af1d8d431b3843cf818d63b
I20260926 14:39:41.037433 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:41.026001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:52543 (local address 127.16.90.254:43913)
0926 14:39:41.026126 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:41.026130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:41.026967 (+ 837us) server_negotiation.cc:407] Connection header received
0926 14:39:41.028302 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:41.028305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:41.028358 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:41.028431 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:41.029531 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:41.030068 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:41.031143 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:41.031359 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:41.031445 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:41.033922 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:41.033925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:41.033927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:41.033950 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:41.035526 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:41.036005 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:41.036009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:41.036012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:41.036110 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:41.036408 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:41.036411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:41.036413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:41.036633 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:41.036724 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:41.037146 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 14:39:41.037276 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:39:41.038432 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:41.026395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:43913 (local address 127.16.90.194:52543)
0926 14:39:41.026832 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:41.026872 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:41.028101 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:41.028606 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:41.028613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:41.028940 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:41.029398 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:41.029411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:41.030188 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:41.030191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:41.031029 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:41.031038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:41.031490 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:41.031493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:41.031641 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:41.032260 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:41.032275 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:41.033674 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:41.035627 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:41.035632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:41.035646 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:41.035909 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:41.036197 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:41.036199 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:41.036200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:41.036326 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:41.036716 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:41.036720 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:41.036931 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 14:39:41.037071 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 14:39:41.037421 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":111,"threads_started":1}
I20260926 14:39:41.039601 20131 heartbeater.cc:347] Connected to a master server at 127.16.90.254:43913
I20260926 14:39:41.039858 20131 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:41.040362 20131 heartbeater.cc:511] Master 127.16.90.254:43913 requested a full tablet report, sending...
I20260926 14:39:41.041299 19806 ts_manager.cc:194] Registered new tserver with Master: 4412aa84d53548c79a889e6fb3d440da (127.16.90.194:41201)
I20260926 14:39:41.041855 19806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:52543
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:39:41.060745 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43913
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:41.167380 20136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:41.167632 20136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:41.167708 20136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:41.171787 20136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:41.171841 20136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:41.171917 20136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:41.177007 20136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:41.177959 20136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:41.178920 20136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:39:41.188095 20136 server_base.cc:1034] running on GCE node
W20260926 14:39:41.188022 20144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:41.188076 20142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:41.188023 20141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:41.188619 20136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:41.189170 20136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:41.190333 20136 hybrid_clock.cc:648] HybridClock initialized: now 1790433581190313 us; error 31 us; skew 500 ppm
Sep 26 14:39:41 dist-test-slave-x4sn krb5kdc[19761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433581, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:41.193033 20136 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:39:41.193943 20136 webserver.cc:460] Webserver started at http://127.16.90.195:35239/ using document root <none> and password file <none>
I20260926 14:39:41.194451 20136 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:41.194509 20136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:41.194689 20136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:41.196633 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "b59d428926a5408a95017b71012bd6c7"
format_stamp: "Formatted at 2026-09-26 14:39:41 on dist-test-slave-x4sn"
server_key: "8f5a905d29020bcd83757c6046fb66e2"
server_key_iv: "17b89542c477d7fe7557057d4f778e8c"
server_key_version: "encryptionkey@0"
I20260926 14:39:41.197145 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "b59d428926a5408a95017b71012bd6c7"
format_stamp: "Formatted at 2026-09-26 14:39:41 on dist-test-slave-x4sn"
server_key: "8f5a905d29020bcd83757c6046fb66e2"
server_key_iv: "17b89542c477d7fe7557057d4f778e8c"
server_key_version: "encryptionkey@0"
I20260926 14:39:41.200573 20136 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:41.202822 20151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:41.204054 20136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:41.204162 20136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "b59d428926a5408a95017b71012bd6c7"
format_stamp: "Formatted at 2026-09-26 14:39:41 on dist-test-slave-x4sn"
server_key: "8f5a905d29020bcd83757c6046fb66e2"
server_key_iv: "17b89542c477d7fe7557057d4f778e8c"
server_key_version: "encryptionkey@0"
I20260926 14:39:41.204252 20136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:41.218658 20136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:41.221575 20136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:41.221761 20136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:41.222304 20136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:41.223066 20136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:41.223132 20136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:41.223186 20136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:41.223212 20136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:41.231415 20136 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:35659
I20260926 14:39:41.231447 20264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:35659 every 8 connection(s)
I20260926 14:39:41.232254 20136 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:41.236720 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 20136
I20260926 14:39:41.236828 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:39:41.237073 16747 external_mini_cluster.cc:1428] Setting key a570ba77032821e7a95f564a6cd14cc8
Sep 26 14:39:41 dist-test-slave-x4sn krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433581, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:41.244712 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:41.233910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:53119 (local address 127.16.90.254:43913)
0926 14:39:41.234075 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:41.234080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:41.234897 (+ 817us) server_negotiation.cc:407] Connection header received
0926 14:39:41.236202 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:41.236205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:41.236252 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:41.236323 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:41.237480 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:41.238053 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:41.239430 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:41.239642 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:41.239742 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:41.241915 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:41.241918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:41.241920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:41.241943 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:41.243170 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:41.243553 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:41.243555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:41.243556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:41.243621 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:41.243890 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:41.243892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:41.243893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:41.244049 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:41.244120 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:41.244505 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 14:39:41.244597 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 14:39:41.245443 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:41.234290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:43913 (local address 127.16.90.195:53119)
0926 14:39:41.234748 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:41.234791 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:41.235986 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:41.236469 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:41.236475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:41.236836 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:41.237330 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:41.237348 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:41.238175 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:41.238178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:41.239333 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:41.239342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:41.239774 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:41.239777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:41.239906 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:41.240389 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:41.240403 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:41.241768 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:41.243252 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:41.243255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:41.243265 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:41.243483 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:41.243700 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:41.243702 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:41.243704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:41.243824 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:41.244125 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:41.244129 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:41.244330 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 14:39:41.244472 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 14:39:41.244710 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":122,"threads_started":1}
I20260926 14:39:41.246441 20265 heartbeater.cc:347] Connected to a master server at 127.16.90.254:43913
I20260926 14:39:41.246657 20265 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:41.247114 20265 heartbeater.cc:511] Master 127.16.90.254:43913 requested a full tablet report, sending...
I20260926 14:39:41.248100 19806 ts_manager.cc:194] Registered new tserver with Master: b59d428926a5408a95017b71012bd6c7 (127.16.90.195:35659)
I20260926 14:39:41.248621 19806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:53119
I20260926 14:39:41.250579 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:39:41 dist-test-slave-x4sn krb5kdc[19761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433581, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 14:39:41 dist-test-slave-x4sn krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433581, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:41.274194 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:41.266175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.16.90.254:43913)
0926 14:39:41.266334 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:41.266339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:41.266469 (+ 130us) server_negotiation.cc:407] Connection header received
0926 14:39:41.266587 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:41.266589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:41.266634 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:41.266707 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:41.267364 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:41.268022 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:41.268833 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:41.269048 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:41.269144 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:41.271355 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:41.271358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:41.271360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:41.271387 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:41.272779 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:41.273105 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:41.273136 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:41.273137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:41.273209 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:41.273386 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:41.273388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:41.273389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:41.273587 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:41.273687 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:41.273932 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 14:39:41.274023 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 14:39:41.274623 19806 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48814
I20260926 14:39:41.275427 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 19868
I20260926 14:39:41.280441 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 20002
I20260926 14:39:41.284687 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 20136
I20260926 14:39:41.288795 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 19774
2026-09-26T14:39:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1982 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:39:41 dist-test-slave-x4sn krb5kdc[20277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:39:41 dist-test-slave-x4sn krb5kdc[20277](info): set up 2 sockets
Sep 26 14:39:41 dist-test-slave-x4sn krb5kdc[20277](info): commencing operation
krb5kdc: starting...
W20260926 14:39:42.241042 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:39:42 dist-test-slave-x4sn krb5kdc[20277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:39:42.354898 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:43281
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:42.459831 20290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:42.460079 20290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:42.460134 20290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:42.464018 20290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:42.464071 20290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:42.464097 20290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:42.464116 20290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:42.464138 20290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:42.469336 20290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:43281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:42.470319 20290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:42.471284 20290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:42.479326 20296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:42.479470 20290 server_base.cc:1034] running on GCE node
W20260926 14:39:42.479346 20295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:42.479346 20298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:42.480149 20290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:42.481202 20290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:42.482369 20290 hybrid_clock.cc:648] HybridClock initialized: now 1790433582482354 us; error 28 us; skew 500 ppm
Sep 26 14:39:42 dist-test-slave-x4sn krb5kdc[20277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:42.484948 20290 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:39:42.485776 20290 webserver.cc:460] Webserver started at http://127.16.90.254:41737/ using document root <none> and password file <none>
I20260926 14:39:42.486289 20290 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:42.486351 20290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:42.486538 20290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:42.488548 20290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "0a3b2cbf333d41d28920cb2de7933f2a"
format_stamp: "Formatted at 2026-09-26 14:39:42 on dist-test-slave-x4sn"
server_key: "d5f1817a21e543e249caf5a44aee3483"
server_key_iv: "253a160db1c4dc68abe4944e4a005124"
server_key_version: "encryptionkey@0"
I20260926 14:39:42.489055 20290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "0a3b2cbf333d41d28920cb2de7933f2a"
format_stamp: "Formatted at 2026-09-26 14:39:42 on dist-test-slave-x4sn"
server_key: "d5f1817a21e543e249caf5a44aee3483"
server_key_iv: "253a160db1c4dc68abe4944e4a005124"
server_key_version: "encryptionkey@0"
I20260926 14:39:42.492465 20290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:42.494711 20305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:42.495950 20290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:42.496054 20290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "0a3b2cbf333d41d28920cb2de7933f2a"
format_stamp: "Formatted at 2026-09-26 14:39:42 on dist-test-slave-x4sn"
server_key: "d5f1817a21e543e249caf5a44aee3483"
server_key_iv: "253a160db1c4dc68abe4944e4a005124"
server_key_version: "encryptionkey@0"
I20260926 14:39:42.496140 20290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:42.509855 20290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:42.512852 20290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:42.513019 20290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:42.519348 20290 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:43281
I20260926 14:39:42.519410 20357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:43281 every 8 connection(s)
I20260926 14:39:42.520298 20290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:42.522886 20358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:42.528395 20358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a: Bootstrap starting.
I20260926 14:39:42.530501 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 20290
I20260926 14:39:42.530622 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:42.530764 20358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:42.530926 16747 external_mini_cluster.cc:1428] Setting key ffdbab500bcf69c863e0df8e60c41ea9
I20260926 14:39:42.531555 20358 log.cc:824] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:42.533488 20358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a: No bootstrap required, opened a new log
I20260926 14:39:42.536733 20358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3b2cbf333d41d28920cb2de7933f2a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43281 } }
I20260926 14:39:42.536931 20358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:42.536996 20358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3b2cbf333d41d28920cb2de7933f2a, State: Initialized, Role: FOLLOWER
I20260926 14:39:42.537397 20358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3b2cbf333d41d28920cb2de7933f2a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43281 } }
I20260926 14:39:42.537520 20358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:42.537580 20358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:42.537667 20358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:42.538409 20358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3b2cbf333d41d28920cb2de7933f2a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43281 } }
Sep 26 14:39:42 dist-test-slave-x4sn krb5kdc[20277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:42.538697 20358 leader_election.cc:302] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a3b2cbf333d41d28920cb2de7933f2a; no voters:
I20260926 14:39:42.539003 20358 leader_election.cc:288] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:42.539080 20363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:42.539323 20363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [term 1 LEADER]: Becoming Leader. State: Replica: 0a3b2cbf333d41d28920cb2de7933f2a, State: Running, Role: LEADER
I20260926 14:39:42.539660 20363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3b2cbf333d41d28920cb2de7933f2a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43281 } }
I20260926 14:39:42.540266 20358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:42.541136 20364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a3b2cbf333d41d28920cb2de7933f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3b2cbf333d41d28920cb2de7933f2a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43281 } } }
I20260926 14:39:42.541342 20364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:42.541610 20364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a3b2cbf333d41d28920cb2de7933f2a. Latest consensus state: current_term: 1 leader_uuid: "0a3b2cbf333d41d28920cb2de7933f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3b2cbf333d41d28920cb2de7933f2a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43281 } } }
I20260926 14:39:42.541759 20364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:42.542460 20372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:42.544584 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:42.532032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.16.90.254:43281)
0926 14:39:42.532381 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:42.532402 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:42.532444 (+ 42us) server_negotiation.cc:407] Connection header received
0926 14:39:42.533062 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:42.533078 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:42.533387 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:42.533690 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:42.534468 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:42.535426 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:42.536289 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:42.536748 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:42.536890 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:42.539332 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:42.539338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:42.539353 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:42.539381 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:42.541331 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:42.541860 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:42.541867 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:42.541872 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:42.542012 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:42.542318 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:42.542322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:42.542324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:42.543008 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:42.543162 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:42.543514 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 14:39:42.543850 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260926 14:39:42.545415 20372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:42.550480 20372 catalog_manager.cc:1347] Generated new cluster ID: 03f7b7c1b57847d4824c5acf8c673975
I20260926 14:39:42.550547 20372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:42.568744 20372 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:42.569948 20372 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:42.586239 20372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a3b2cbf333d41d28920cb2de7933f2a: Generated new TSK 0
I20260926 14:39:42.587030 20372 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:39:42.612504 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43281
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:42.717823 20384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:42.718052 20384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:42.718091 20384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:42.722049 20384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:42.722105 20384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:42.722158 20384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:42.727231 20384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:42.728245 20384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:42.729188 20384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:39:42.738996 20384 server_base.cc:1034] running on GCE node
W20260926 14:39:42.739006 20392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:42.739018 20389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:42.739018 20390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:42.739748 20384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:42.740325 20384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:42.741473 20384 hybrid_clock.cc:648] HybridClock initialized: now 1790433582741430 us; error 53 us; skew 500 ppm
Sep 26 14:39:42 dist-test-slave-x4sn krb5kdc[20277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:42.743973 20384 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:39:42.744784 20384 webserver.cc:460] Webserver started at http://127.16.90.193:36583/ using document root <none> and password file <none>
I20260926 14:39:42.745324 20384 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:42.745385 20384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:42.745571 20384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:42.747525 20384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "2ff3ca7783ba433382fd3910e3ec9088"
format_stamp: "Formatted at 2026-09-26 14:39:42 on dist-test-slave-x4sn"
server_key: "8a0baeff42df52104213587f7ef3232a"
server_key_iv: "9bf5b8b0f139cbd28e2dae32561c8a14"
server_key_version: "encryptionkey@0"
I20260926 14:39:42.748078 20384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "2ff3ca7783ba433382fd3910e3ec9088"
format_stamp: "Formatted at 2026-09-26 14:39:42 on dist-test-slave-x4sn"
server_key: "8a0baeff42df52104213587f7ef3232a"
server_key_iv: "9bf5b8b0f139cbd28e2dae32561c8a14"
server_key_version: "encryptionkey@0"
I20260926 14:39:42.751539 20384 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:39:42.753847 20399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:42.755007 20384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:42.755102 20384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "2ff3ca7783ba433382fd3910e3ec9088"
format_stamp: "Formatted at 2026-09-26 14:39:42 on dist-test-slave-x4sn"
server_key: "8a0baeff42df52104213587f7ef3232a"
server_key_iv: "9bf5b8b0f139cbd28e2dae32561c8a14"
server_key_version: "encryptionkey@0"
I20260926 14:39:42.755221 20384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:42.782131 20384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:42.785377 20384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:42.785561 20384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:42.786163 20384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:42.786981 20384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:42.787029 20384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:42.787074 20384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:42.787103 20384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:42.795305 20384 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:39091
I20260926 14:39:42.795334 20512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:39091 every 8 connection(s)
I20260926 14:39:42.796211 20384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:39:42.797860 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 20384
I20260926 14:39:42.797991 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:39:42.798303 16747 external_mini_cluster.cc:1428] Setting key a02184d568f5783a6839725554d90900
Sep 26 14:39:42 dist-test-slave-x4sn krb5kdc[20277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
I20260926 14:39:42.810735 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:42.798052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:46145 (local address 127.16.90.254:43281)
0926 14:39:42.798192 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:42.798197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:42.799105 (+ 908us) server_negotiation.cc:407] Connection header received
0926 14:39:42.800939 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:42.800941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:42.800983 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:42.801041 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:42.802402 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:42.803112 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:42.804302 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:42.804675 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:42.804811 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:42.807220 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:42.807223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:42.807225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:42.807250 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:42.808670 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:42.809293 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:42.809295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:42.809297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:42.809375 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:42.809716 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:42.809718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:42.809719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:42.809889 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:42.809957 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:42.810427 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 14:39:42.810601 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:39:42.811517 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:42.798409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:43281 (local address 127.16.90.193:46145)
0926 14:39:42.798942 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:42.799012 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:42.800695 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:42.801215 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:42.801224 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:42.801737 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:42.802253 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:42.802267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:42.803237 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:42.803240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:42.804184 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:42.804194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:42.804834 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:42.804836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:42.804978 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:42.805511 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:42.805526 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:42.807043 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:42.808788 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:42.808793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:42.808806 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:42.809187 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:42.809478 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:42.809481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:42.809482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:42.809624 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:42.809962 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:42.809967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:42.810203 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 14:39:42.810355 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 14:39:42.810628 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":116,"threads_started":1}
I20260926 14:39:42.812722 20513 heartbeater.cc:347] Connected to a master server at 127.16.90.254:43281
I20260926 14:39:42.812989 20513 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:42.813555 20513 heartbeater.cc:511] Master 127.16.90.254:43281 requested a full tablet report, sending...
I20260926 14:39:42.815040 20322 ts_manager.cc:194] Registered new tserver with Master: 2ff3ca7783ba433382fd3910e3ec9088 (127.16.90.193:39091)
I20260926 14:39:42.816170 20322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:46145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:39:42.824698 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43281
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:42.930840 20518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:42.931100 20518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:42.931142 20518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:42.935108 20518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:42.935197 20518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:42.935278 20518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:42.940615 20518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:42.941682 20518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:42.942710 20518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:42.952065 20523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:42.952066 20524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:42.952065 20526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:42.952288 20518 server_base.cc:1034] running on GCE node
I20260926 14:39:42.952685 20518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:42.953295 20518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:42.954454 20518 hybrid_clock.cc:648] HybridClock initialized: now 1790433582954433 us; error 33 us; skew 500 ppm
Sep 26 14:39:42 dist-test-slave-x4sn krb5kdc[20277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:42.957048 20518 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:39:42.957926 20518 webserver.cc:460] Webserver started at http://127.16.90.194:33677/ using document root <none> and password file <none>
I20260926 14:39:42.958475 20518 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:42.958544 20518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:42.958760 20518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:42.960920 20518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "a48246bf25a140a2b6b7ff7fb9c069c4"
format_stamp: "Formatted at 2026-09-26 14:39:42 on dist-test-slave-x4sn"
server_key: "8f9781523ab2de0b1bc2e869f5d1476b"
server_key_iv: "5f1c7e14ffb269a90ebebbd628ba4edb"
server_key_version: "encryptionkey@0"
I20260926 14:39:42.961473 20518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "a48246bf25a140a2b6b7ff7fb9c069c4"
format_stamp: "Formatted at 2026-09-26 14:39:42 on dist-test-slave-x4sn"
server_key: "8f9781523ab2de0b1bc2e869f5d1476b"
server_key_iv: "5f1c7e14ffb269a90ebebbd628ba4edb"
server_key_version: "encryptionkey@0"
I20260926 14:39:42.965169 20518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:39:42.967578 20533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:42.968791 20518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:42.968897 20518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "a48246bf25a140a2b6b7ff7fb9c069c4"
format_stamp: "Formatted at 2026-09-26 14:39:42 on dist-test-slave-x4sn"
server_key: "8f9781523ab2de0b1bc2e869f5d1476b"
server_key_iv: "5f1c7e14ffb269a90ebebbd628ba4edb"
server_key_version: "encryptionkey@0"
I20260926 14:39:42.968994 20518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:42.995273 20518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:42.998435 20518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:42.998617 20518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:42.999238 20518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:43.000083 20518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:43.000130 20518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:43.000193 20518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:43.000257 20518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:43.008554 20518 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:36775
I20260926 14:39:43.008579 20646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:36775 every 8 connection(s)
I20260926 14:39:43.009435 20518 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:39:43.010761 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 20518
I20260926 14:39:43.010890 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:39:43.011202 16747 external_mini_cluster.cc:1428] Setting key a5bdab781098f42131e8c243dffb6d41
Sep 26 14:39:43 dist-test-slave-x4sn krb5kdc[20277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
I20260926 14:39:43.024061 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:43.011314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:36667 (local address 127.16.90.254:43281)
0926 14:39:43.011429 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:43.011432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:43.012495 (+ 1063us) server_negotiation.cc:407] Connection header received
0926 14:39:43.013872 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:43.013875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:43.013930 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:43.014022 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:43.015547 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:43.016333 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:43.017775 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:43.017996 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:43.018077 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:43.020807 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:43.020810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:43.020812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:43.020836 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:43.022064 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:43.022493 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:43.022495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:43.022496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:43.022561 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:43.022849 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:43.022851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:43.022853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:43.023033 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:43.023184 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:43.023759 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 14:39:43.023896 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:39:43.024864 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:43.011840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:43281 (local address 127.16.90.194:36667)
0926 14:39:43.012347 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:43.012398 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:43.013655 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:43.014246 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:43.014255 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:43.014719 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:43.015402 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:43.015419 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:43.016442 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:43.016444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:43.017658 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:43.017669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:43.018142 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:43.018147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:43.018367 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:43.018953 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:43.018974 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:43.020637 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:43.022163 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:43.022169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:43.022187 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:43.022419 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:43.022656 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:43.022657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:43.022659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:43.022781 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:43.023142 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:43.023146 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:43.023470 (+ 324us) client_negotiation.cc:769] Sending connection context
0926 14:39:43.023661 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 14:39:43.023978 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260926 14:39:43.026005 20647 heartbeater.cc:347] Connected to a master server at 127.16.90.254:43281
I20260926 14:39:43.026239 20647 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:43.026722 20647 heartbeater.cc:511] Master 127.16.90.254:43281 requested a full tablet report, sending...
I20260926 14:39:43.027778 20322 ts_manager.cc:194] Registered new tserver with Master: a48246bf25a140a2b6b7ff7fb9c069c4 (127.16.90.194:36775)
I20260926 14:39:43.028332 20322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:36667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:39:43.038939 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43281
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:43.147821 20652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:43.148109 20652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:43.148170 20652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:43.152217 20652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:43.152280 20652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:43.152364 20652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:43.157517 20652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:43.158608 20652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:43.159648 20652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:43.168856 20657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:43.168875 20660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:43.168962 20652 server_base.cc:1034] running on GCE node
W20260926 14:39:43.168856 20658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:43.169466 20652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:43.170091 20652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:43.171258 20652 hybrid_clock.cc:648] HybridClock initialized: now 1790433583171234 us; error 41 us; skew 500 ppm
Sep 26 14:39:43 dist-test-slave-x4sn krb5kdc[20277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433583, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:43.174065 20652 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:39:43.174899 20652 webserver.cc:460] Webserver started at http://127.16.90.195:45467/ using document root <none> and password file <none>
I20260926 14:39:43.175429 20652 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:43.175487 20652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:43.175690 20652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:43.177675 20652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "74d69d4f36e742d18caf1c3faf19ef12"
format_stamp: "Formatted at 2026-09-26 14:39:43 on dist-test-slave-x4sn"
server_key: "2298553437dba3d783c8ba88b705dc7c"
server_key_iv: "fa19ae01e5e1ab150ba6281feeaf3dca"
server_key_version: "encryptionkey@0"
I20260926 14:39:43.178164 20652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "74d69d4f36e742d18caf1c3faf19ef12"
format_stamp: "Formatted at 2026-09-26 14:39:43 on dist-test-slave-x4sn"
server_key: "2298553437dba3d783c8ba88b705dc7c"
server_key_iv: "fa19ae01e5e1ab150ba6281feeaf3dca"
server_key_version: "encryptionkey@0"
I20260926 14:39:43.181641 20652 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:39:43.183959 20667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:43.185168 20652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:39:43.185276 20652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "74d69d4f36e742d18caf1c3faf19ef12"
format_stamp: "Formatted at 2026-09-26 14:39:43 on dist-test-slave-x4sn"
server_key: "2298553437dba3d783c8ba88b705dc7c"
server_key_iv: "fa19ae01e5e1ab150ba6281feeaf3dca"
server_key_version: "encryptionkey@0"
I20260926 14:39:43.185375 20652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:43.202569 20652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:43.205430 20652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:43.205602 20652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:43.206197 20652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:43.207011 20652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:43.207057 20652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:43.207100 20652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:43.207127 20652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:43.215324 20652 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:35659
I20260926 14:39:43.215356 20780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:35659 every 8 connection(s)
I20260926 14:39:43.216213 20652 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:43.225032 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 20652
I20260926 14:39:43.225132 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
Sep 26 14:39:43 dist-test-slave-x4sn krb5kdc[20277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433583, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:43.225405 16747 external_mini_cluster.cc:1428] Setting key 08b27f1e1df189fda9e290a29d2ff656
I20260926 14:39:43.229046 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:43.217884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:58025 (local address 127.16.90.254:43281)
0926 14:39:43.218020 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:43.218023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:43.218835 (+ 812us) server_negotiation.cc:407] Connection header received
0926 14:39:43.220159 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:43.220163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:43.220212 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:43.220285 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:43.221420 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:43.222020 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:43.223257 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:43.223478 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:43.223561 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:43.225840 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:43.225842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:43.225844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:43.225869 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:43.227136 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:43.227604 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:43.227607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:43.227608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:43.227719 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:43.228024 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:43.228027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:43.228029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:43.228212 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:43.228297 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:43.228771 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 14:39:43.228891 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:39:43.229830 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:43.218253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:43281 (local address 127.16.90.195:58025)
0926 14:39:43.218678 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:43.218721 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:43.219964 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:43.220435 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:43.220442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:43.220780 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:43.221264 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:43.221278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:43.222128 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:43.222131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:43.223140 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:43.223149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:43.223594 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:43.223597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:43.223751 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:43.224271 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:43.224286 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:43.225682 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:43.227251 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:43.227256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:43.227271 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:43.227510 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:43.227799 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:43.227802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:43.227804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:43.227938 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:43.228307 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:43.228313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:43.228545 (+ 232us) client_negotiation.cc:769] Sending connection context
0926 14:39:43.228714 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 14:39:43.229024 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260926 14:39:43.230777 20781 heartbeater.cc:347] Connected to a master server at 127.16.90.254:43281
I20260926 14:39:43.230991 20781 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:43.231429 20781 heartbeater.cc:511] Master 127.16.90.254:43281 requested a full tablet report, sending...
I20260926 14:39:43.232373 20322 ts_manager.cc:194] Registered new tserver with Master: 74d69d4f36e742d18caf1c3faf19ef12 (127.16.90.195:35659)
I20260926 14:39:43.232894 20322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:58025
I20260926 14:39:43.238610 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:39:43 dist-test-slave-x4sn krb5kdc[20277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:39:43 dist-test-slave-x4sn krb5kdc[20277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:39:43.262950 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:43.253973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.16.90.193:39091)
0926 14:39:43.254224 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:43.254240 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:43.254264 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:39:43.254336 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:43.254341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:43.254527 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:43.254646 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:43.255228 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:43.256050 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:43.256946 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:43.257326 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:43.257413 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:43.259905 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:43.259912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:43.259924 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:43.259947 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:43.261491 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:43.261882 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:43.261887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:43.261888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:43.261961 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:43.262147 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:43.262148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:43.262149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:43.262395 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:43.262567 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:43.262687 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 14:39:43.262805 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":61,"threads_started":1}
Sep 26 14:39:43 dist-test-slave-x4sn krb5kdc[20277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:43.272799 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:43.265630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.16.90.254:43281)
0926 14:39:43.265783 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:43.265787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:43.265800 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:39:43.265883 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:43.265885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:43.265923 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:43.265984 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:43.266490 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:43.266978 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:43.267757 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:43.268021 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:43.268110 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:43.270206 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:43.270209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:43.270211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:43.270235 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:43.271497 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:43.271863 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:43.271866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:43.271867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:43.271939 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:43.272120 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:43.272121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:43.272123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:43.272316 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:43.272413 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:43.272577 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 14:39:43.272662 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 14:39:43.273263 20322 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54042
I20260926 14:39:43.273856 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 20384
I20260926 14:39:43.279414 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 20518
I20260926 14:39:43.283908 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 20652
I20260926 14:39:43.288327 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 20290
2026-09-26T14:39:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1999 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:39:43 dist-test-slave-x4sn krb5kdc[20793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:39:43 dist-test-slave-x4sn krb5kdc[20793](info): set up 2 sockets
Sep 26 14:39:43 dist-test-slave-x4sn krb5kdc[20793](info): commencing operation
krb5kdc: starting...
W20260926 14:39:44.245139 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:39:44 dist-test-slave-x4sn krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:39:44.363575 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:40689
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:40689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:44.469591 20806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:44.469844 20806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:44.469884 20806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:44.473863 20806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:44.473917 20806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:44.473934 20806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:44.473949 20806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:44.473963 20806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:44.478991 20806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:40689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:40689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:44.479983 20806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:44.480938 20806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:44.489161 20812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:44.489187 20811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:44.489172 20814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:44.489313 20806 server_base.cc:1034] running on GCE node
I20260926 14:39:44.490000 20806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:44.491171 20806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:44.492349 20806 hybrid_clock.cc:648] HybridClock initialized: now 1790433584492322 us; error 39 us; skew 500 ppm
Sep 26 14:39:44 dist-test-slave-x4sn krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:44.499257 20806 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:39:44.500043 20806 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:44.500118 20806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:44.500319 20806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:44.502337 20806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "bf6d554d1890492e80ecce432a5f7454"
format_stamp: "Formatted at 2026-09-26 14:39:44 on dist-test-slave-x4sn"
server_key: "d6b3eb9e6937abd2bd15bee314ed3cd8"
server_key_iv: "96c75c45666b70af2cf5ffbd6781f6ec"
server_key_version: "encryptionkey@0"
I20260926 14:39:44.502877 20806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "bf6d554d1890492e80ecce432a5f7454"
format_stamp: "Formatted at 2026-09-26 14:39:44 on dist-test-slave-x4sn"
server_key: "d6b3eb9e6937abd2bd15bee314ed3cd8"
server_key_iv: "96c75c45666b70af2cf5ffbd6781f6ec"
server_key_version: "encryptionkey@0"
I20260926 14:39:44.506342 20806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 14:39:44.508667 20820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:44.509838 20806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:44.509938 20806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "bf6d554d1890492e80ecce432a5f7454"
format_stamp: "Formatted at 2026-09-26 14:39:44 on dist-test-slave-x4sn"
server_key: "d6b3eb9e6937abd2bd15bee314ed3cd8"
server_key_iv: "96c75c45666b70af2cf5ffbd6781f6ec"
server_key_version: "encryptionkey@0"
I20260926 14:39:44.510015 20806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:44.536584 20806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:44.539623 20806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:44.539806 20806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:44.546351 20806 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:40689
I20260926 14:39:44.546586 20872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:40689 every 8 connection(s)
I20260926 14:39:44.546964 20806 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:44.549394 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 20806
I20260926 14:39:44.549505 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:44.549629 20873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:44.549801 16747 external_mini_cluster.cc:1428] Setting key fc99c1b4431d81f8973f94c93ec716f2
I20260926 14:39:44.556231 20873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454: Bootstrap starting.
Sep 26 14:39:44 dist-test-slave-x4sn krb5kdc[20793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:44.558729 20873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:44.559566 20873 log.cc:824] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:44.561614 20873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454: No bootstrap required, opened a new log
I20260926 14:39:44.563545 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:44.551113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.16.90.254:40689)
0926 14:39:44.551487 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:44.551502 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:44.551537 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:39:44.552185 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:44.552201 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:44.552519 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:44.552800 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:44.553645 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:44.554641 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:44.555778 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:44.556241 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:44.556371 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:44.558918 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:44.558925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:44.558943 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:44.558975 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:44.560660 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:44.561091 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:44.561096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:44.561102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:44.561231 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:44.561441 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:44.561444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:44.561446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:44.561935 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:44.562113 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:44.562365 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 14:39:44.562639 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260926 14:39:44.565407 20873 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6d554d1890492e80ecce432a5f7454" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40689 } }
I20260926 14:39:44.565603 20873 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:44.565640 20873 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf6d554d1890492e80ecce432a5f7454, State: Initialized, Role: FOLLOWER
I20260926 14:39:44.565969 20873 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6d554d1890492e80ecce432a5f7454" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40689 } }
I20260926 14:39:44.566072 20873 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:44.566108 20873 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:44.566165 20873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:44.566919 20873 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6d554d1890492e80ecce432a5f7454" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40689 } }
I20260926 14:39:44.567199 20873 leader_election.cc:302] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf6d554d1890492e80ecce432a5f7454; no voters:
I20260926 14:39:44.567426 20873 leader_election.cc:288] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:44.567516 20878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:44.567778 20878 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [term 1 LEADER]: Becoming Leader. State: Replica: bf6d554d1890492e80ecce432a5f7454, State: Running, Role: LEADER
I20260926 14:39:44.568073 20878 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6d554d1890492e80ecce432a5f7454" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40689 } }
I20260926 14:39:44.568524 20873 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:44.569833 20880 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf6d554d1890492e80ecce432a5f7454. Latest consensus state: current_term: 1 leader_uuid: "bf6d554d1890492e80ecce432a5f7454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6d554d1890492e80ecce432a5f7454" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40689 } } }
I20260926 14:39:44.569944 20880 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:44.570096 20879 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf6d554d1890492e80ecce432a5f7454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6d554d1890492e80ecce432a5f7454" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40689 } } }
I20260926 14:39:44.570421 20879 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454 [sys.catalog]: This master's current role is: LEADER
W20260926 14:39:44.573189 20893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:39:44.573251 20893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:39:44.573321 20892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:44.573973 20892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:44.579077 20892 catalog_manager.cc:1347] Generated new cluster ID: f59a5ba7edad48b4a55c5ed44c235b6b
I20260926 14:39:44.579133 20892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:44.590770 20892 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:44.591639 20892 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:44.596865 20892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454: Generated new TSK 0
I20260926 14:39:44.597627 20892 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:39:44.634379 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40689
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:44.741880 20899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:44.742120 20899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:44.742172 20899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:44.746138 20899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:44.746191 20899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:44.746250 20899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:44.751281 20899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:44.752326 20899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:44.753295 20899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:39:44.762255 20899 server_base.cc:1034] running on GCE node
W20260926 14:39:44.762239 20907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:44.762216 20904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:44.762221 20905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:44.762845 20899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:44.763435 20899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:44.764585 20899 hybrid_clock.cc:648] HybridClock initialized: now 1790433584764558 us; error 43 us; skew 500 ppm
Sep 26 14:39:44 dist-test-slave-x4sn krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:44.767162 20899 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:39:44.767853 20899 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:44.767915 20899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:44.768105 20899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:44.770202 20899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "efdcdf2da82845eba2550d93811e817f"
format_stamp: "Formatted at 2026-09-26 14:39:44 on dist-test-slave-x4sn"
server_key: "c224b7a3241d9cae08edd40f6c144bec"
server_key_iv: "30a52f00cbd8ee955dbf10f099e299aa"
server_key_version: "encryptionkey@0"
I20260926 14:39:44.770758 20899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "efdcdf2da82845eba2550d93811e817f"
format_stamp: "Formatted at 2026-09-26 14:39:44 on dist-test-slave-x4sn"
server_key: "c224b7a3241d9cae08edd40f6c144bec"
server_key_iv: "30a52f00cbd8ee955dbf10f099e299aa"
server_key_version: "encryptionkey@0"
I20260926 14:39:44.774379 20899 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:39:44.776762 20913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:44.777933 20899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:39:44.778030 20899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "efdcdf2da82845eba2550d93811e817f"
format_stamp: "Formatted at 2026-09-26 14:39:44 on dist-test-slave-x4sn"
server_key: "c224b7a3241d9cae08edd40f6c144bec"
server_key_iv: "30a52f00cbd8ee955dbf10f099e299aa"
server_key_version: "encryptionkey@0"
I20260926 14:39:44.778137 20899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:44.788895 20899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:44.791885 20899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:44.792050 20899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:44.792585 20899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:44.793345 20899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:44.793390 20899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:44.793431 20899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:44.793460 20899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:44.801198 20899 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:44605
I20260926 14:39:44.801223 21026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:44605 every 8 connection(s)
I20260926 14:39:44.801762 20899 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:39:44.809933 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 20899
I20260926 14:39:44.810060 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:39:44.810328 16747 external_mini_cluster.cc:1428] Setting key e80e9d890e37b68422c7fe25463e61c6
Sep 26 14:39:44 dist-test-slave-x4sn krb5kdc[20793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:44.815488 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:44.803310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:55191 (local address 127.16.90.254:40689)
0926 14:39:44.803422 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:44.803425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:44.804733 (+ 1308us) server_negotiation.cc:407] Connection header received
0926 14:39:44.805588 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:44.805591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:44.805641 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:44.805720 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:44.806895 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:44.807486 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:44.808773 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:44.809031 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:44.809127 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:44.811631 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:44.811635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:44.811637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:44.811691 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:44.813201 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:44.813844 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:44.813847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:44.813848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:44.813927 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:44.814356 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:44.814359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:44.814360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:44.814543 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:44.814639 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:44.815167 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 14:39:44.815333 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:39:44.816319 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:44.803789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40689 (local address 127.16.90.193:55191)
0926 14:39:44.804585 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:44.804625 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:44.805362 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:44.805867 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:44.805873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:44.806215 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:44.806691 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:44.806705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:44.807646 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:44.807649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:44.808619 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:44.808639 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:44.809204 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:44.809209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:44.809361 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:44.809961 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:44.809978 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:44.811462 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:44.813332 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:44.813339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:44.813357 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:44.813751 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:44.814008 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:44.814011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:44.814013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:44.814257 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:44.814659 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:44.814664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:44.814923 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 14:39:44.815086 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 14:39:44.815368 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":135,"threads_started":1}
I20260926 14:39:44.817524 21027 heartbeater.cc:347] Connected to a master server at 127.16.90.254:40689
I20260926 14:39:44.817754 21027 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:44.818169 21027 heartbeater.cc:511] Master 127.16.90.254:40689 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
I20260926 14:39:44.819564 20837 ts_manager.cc:194] Registered new tserver with Master: efdcdf2da82845eba2550d93811e817f (127.16.90.193:44605)
I20260926 14:39:44.820778 20837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:55191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:39:44.836297 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40689
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:44.945247 21032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:44.945505 21032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:44.945549 21032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:44.949443 21032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:44.949496 21032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:44.949554 21032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:44.954717 21032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:44.955987 21032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:44.957309 21032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:44.967315 21037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:44.967438 21040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:44.967383 21038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:44.967517 21032 server_base.cc:1034] running on GCE node
I20260926 14:39:44.968075 21032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:44.968667 21032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:44.969810 21032 hybrid_clock.cc:648] HybridClock initialized: now 1790433584969792 us; error 31 us; skew 500 ppm
Sep 26 14:39:44 dist-test-slave-x4sn krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:44.972458 21032 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:39:44.973142 21032 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:44.973209 21032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:44.973397 21032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:44.975322 21032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "84dd67c2f6e0474da6fb658351e75f51"
format_stamp: "Formatted at 2026-09-26 14:39:44 on dist-test-slave-x4sn"
server_key: "8334e439d2aeeb5c8b1cf22f9c735f3e"
server_key_iv: "88300ba684a81555822eab0c70c214fc"
server_key_version: "encryptionkey@0"
I20260926 14:39:44.975871 21032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "84dd67c2f6e0474da6fb658351e75f51"
format_stamp: "Formatted at 2026-09-26 14:39:44 on dist-test-slave-x4sn"
server_key: "8334e439d2aeeb5c8b1cf22f9c735f3e"
server_key_iv: "88300ba684a81555822eab0c70c214fc"
server_key_version: "encryptionkey@0"
I20260926 14:39:44.979321 21032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 14:39:44.981680 21046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:44.982885 21032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:44.982995 21032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "84dd67c2f6e0474da6fb658351e75f51"
format_stamp: "Formatted at 2026-09-26 14:39:44 on dist-test-slave-x4sn"
server_key: "8334e439d2aeeb5c8b1cf22f9c735f3e"
server_key_iv: "88300ba684a81555822eab0c70c214fc"
server_key_version: "encryptionkey@0"
I20260926 14:39:44.983089 21032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:44.996115 21032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:44.999158 21032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:44.999331 21032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:44.999943 21032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:45.000746 21032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:45.000790 21032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:45.000842 21032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:45.000870 21032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:45.009279 21032 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:34511
I20260926 14:39:45.009301 21159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:34511 every 8 connection(s)
I20260926 14:39:45.009828 21032 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:39:45.011857 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 21032
I20260926 14:39:45.011992 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:39:45.012320 16747 external_mini_cluster.cc:1428] Setting key a91ece13f884c176a136d805b6597514
Sep 26 14:39:45 dist-test-slave-x4sn krb5kdc[20793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433584, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
I20260926 14:39:45.023710 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.011529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:53881 (local address 127.16.90.254:40689)
0926 14:39:45.011735 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:45.011738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:45.012987 (+ 1249us) server_negotiation.cc:407] Connection header received
0926 14:39:45.013986 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:45.013988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:45.014028 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:45.014077 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:45.015574 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.016381 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.017557 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.017905 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.018012 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.020448 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:45.020452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:45.020454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:45.020481 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:45.021939 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:45.022392 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:45.022395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:45.022397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:45.022470 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:45.022754 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:45.022757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:45.022758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:45.022928 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:45.023000 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:45.023468 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 14:39:45.023563 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
I20260926 14:39:45.024711 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.011987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40689 (local address 127.16.90.194:53881)
0926 14:39:45.012829 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:45.012881 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:45.013779 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:45.014284 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:45.014293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:45.014739 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:45.015400 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.015422 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.016518 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.016521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.017428 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.017437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.018056 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.018059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.018232 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.018849 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:45.018864 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:45.020282 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:45.022039 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:45.022044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:45.022055 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:45.022307 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:45.022538 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:45.022540 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:45.022541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:45.022674 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:45.022999 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:45.023003 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:45.023250 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 14:39:45.023439 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 14:39:45.023726 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":138,"threads_started":1}
I20260926 14:39:45.025835 21160 heartbeater.cc:347] Connected to a master server at 127.16.90.254:40689
I20260926 14:39:45.026094 21160 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:45.026597 21160 heartbeater.cc:511] Master 127.16.90.254:40689 requested a full tablet report, sending...
I20260926 14:39:45.027623 20837 ts_manager.cc:194] Registered new tserver with Master: 84dd67c2f6e0474da6fb658351e75f51 (127.16.90.194:34511)
I20260926 14:39:45.028385 20837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:53881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:39:45.039240 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40689
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:45.145404 21165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:45.145646 21165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:45.145689 21165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:45.149524 21165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:45.149571 21165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:45.149632 21165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:45.154556 21165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:40689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:45.155467 21165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:45.156410 21165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:45.165103 21170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:45.165109 21171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:45.165117 21173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:45.165391 21165 server_base.cc:1034] running on GCE node
I20260926 14:39:45.165773 21165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:45.166374 21165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:45.167529 21165 hybrid_clock.cc:648] HybridClock initialized: now 1790433585167513 us; error 28 us; skew 500 ppm
Sep 26 14:39:45 dist-test-slave-x4sn krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433585, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:45.170269 21165 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:39:45.170969 21165 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:45.171030 21165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:45.171278 21165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:45.173295 21165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "da605c2b00cb439cbbbffa67c3a94acb"
format_stamp: "Formatted at 2026-09-26 14:39:45 on dist-test-slave-x4sn"
server_key: "c98e39224df21bb577124c208377f0ee"
server_key_iv: "cac932d7a9aa6aeb33af820fcc0e6748"
server_key_version: "encryptionkey@0"
I20260926 14:39:45.173830 21165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "da605c2b00cb439cbbbffa67c3a94acb"
format_stamp: "Formatted at 2026-09-26 14:39:45 on dist-test-slave-x4sn"
server_key: "c98e39224df21bb577124c208377f0ee"
server_key_iv: "cac932d7a9aa6aeb33af820fcc0e6748"
server_key_version: "encryptionkey@0"
I20260926 14:39:45.177544 21165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 14:39:45.179913 21179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:45.181141 21165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:45.181248 21165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "da605c2b00cb439cbbbffa67c3a94acb"
format_stamp: "Formatted at 2026-09-26 14:39:45 on dist-test-slave-x4sn"
server_key: "c98e39224df21bb577124c208377f0ee"
server_key_iv: "cac932d7a9aa6aeb33af820fcc0e6748"
server_key_version: "encryptionkey@0"
I20260926 14:39:45.181339 21165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:45.194622 21165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:45.197664 21165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:45.197813 21165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:45.198326 21165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:45.199108 21165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:45.199153 21165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:45.199190 21165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:45.199210 21165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:45.207475 21165 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:32875
I20260926 14:39:45.207525 21292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:32875 every 8 connection(s)
I20260926 14:39:45.208078 21165 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:45.214828 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 21165
I20260926 14:39:45.214936 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:39:45.215242 16747 external_mini_cluster.cc:1428] Setting key e3a4130867d8319f5d38660aa95ddac4
Sep 26 14:39:45 dist-test-slave-x4sn krb5kdc[20793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433585, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:45.220929 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.209744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:58257 (local address 127.16.90.254:40689)
0926 14:39:45.209868 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:45.209871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:45.210997 (+ 1126us) server_negotiation.cc:407] Connection header received
0926 14:39:45.211860 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:45.211863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:45.211912 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:45.211983 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:45.213132 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.213658 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.214860 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.215194 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.215302 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.217720 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:45.217729 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:45.217731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:45.217757 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:45.219070 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:45.219533 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:45.219536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:45.219537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:45.219619 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:45.219978 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:45.219981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:45.219982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:45.220182 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:45.220269 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:45.220708 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 14:39:45.220809 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:39:45.221735 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.210072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:40689 (local address 127.16.90.195:58257)
0926 14:39:45.210829 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:45.210870 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:45.211610 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:45.212119 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:45.212125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:45.212475 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:45.213003 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.213020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.213793 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.213796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.214728 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.214738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.215344 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.215349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.215497 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.216093 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:45.216115 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:45.217533 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:45.219180 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:45.219184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:45.219196 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:45.219430 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:45.219735 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:45.219738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:45.219739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:45.219871 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:45.220272 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:45.220277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:45.220502 (+ 225us) client_negotiation.cc:769] Sending connection context
0926 14:39:45.220655 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 14:39:45.220902 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
I20260926 14:39:45.222823 21293 heartbeater.cc:347] Connected to a master server at 127.16.90.254:40689
I20260926 14:39:45.223065 21293 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:45.223520 21293 heartbeater.cc:511] Master 127.16.90.254:40689 requested a full tablet report, sending...
I20260926 14:39:45.224522 20837 ts_manager.cc:194] Registered new tserver with Master: da605c2b00cb439cbbbffa67c3a94acb (127.16.90.195:32875)
I20260926 14:39:45.225061 20837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:58257
I20260926 14:39:45.228883 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:39:45.237125 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.230630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.16.90.254:40689)
0926 14:39:45.230744 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:45.230748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:45.230909 (+ 161us) server_negotiation.cc:407] Connection header received
0926 14:39:45.231027 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:45.231030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:45.231067 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:45.231127 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:45.231823 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.232316 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.233117 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.233323 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.233405 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.234604 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:45.234607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:45.234609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:45.234631 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:45.235611 (+ 980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:45.236026 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:45.236029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:45.236031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:45.236102 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:45.236342 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:45.236346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:45.236347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:45.236544 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:45.236674 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:45.236879 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 14:39:45.236983 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:39:45.239539 20837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:39:45.242430 20837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:45.256318 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.251127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.16.90.195:32875)
0926 14:39:45.251704 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:45.251716 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:45.251805 (+ 89us) server_negotiation.cc:407] Connection header received
0926 14:39:45.251877 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:45.251881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:45.252060 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:45.252408 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:45.252520 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.253189 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.254873 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.255703 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.255815 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.255961 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:45.256103 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 14:39:45.256190 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":121,"threads_started":1}
I20260926 14:39:45.256336 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.250956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:32875 (local address 127.0.0.1:44486)
0926 14:39:45.251372 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:45.251383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:45.251456 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:45.252200 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:45.252202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:45.252215 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:45.252283 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.252289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.253308 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.253311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.254742 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.254751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.255815 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.255818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.255925 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.255962 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 14:39:45.256072 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 14:39:45.256193 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":52,"thread_start_us":79,"threads_started":1}
I20260926 14:39:45.256875 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.250261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:34511 (local address 127.0.0.1:40714)
0926 14:39:45.250576 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:45.250635 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:45.250759 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:45.252142 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:45.252146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:45.252164 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:45.252253 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.252260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.253559 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.253562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.254721 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.254730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.256486 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.256490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.256610 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.256630 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:39:45.256707 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 14:39:45.256776 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":23,"thread_start_us":69,"threads_started":1}
I20260926 14:39:45.257308 21306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.250473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:44605 (local address 127.0.0.1:34764)
0926 14:39:45.251100 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:45.251113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:45.251206 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:45.251923 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:45.251929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:45.251951 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:45.252092 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.252100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.254731 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.254734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.255474 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.255483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.256514 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.256517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.257040 (+ 523us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.257063 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:39:45.257126 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:39:45.257191 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":413,"thread_start_us":94,"threads_started":1}
I20260926 14:39:45.257395 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.250887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34764 (local address 127.16.90.193:44605)
0926 14:39:45.251399 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:45.251413 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:45.251434 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:39:45.251508 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:45.251514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:45.251768 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:45.251925 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:45.252220 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.253424 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.255592 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.256392 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.256602 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.256737 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:45.257161 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 14:39:45.257261 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":93,"threads_started":1}
I20260926 14:39:45.257810 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.250393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.16.90.194:34511)
0926 14:39:45.251649 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:45.251662 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:45.251708 (+ 46us) server_negotiation.cc:407] Connection header received
0926 14:39:45.251782 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:45.251787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:45.251962 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:45.252147 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:45.252344 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.253439 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.254841 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.255711 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.255815 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.255962 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:45.257552 (+ 1590us) server_negotiation.cc:299] Negotiation successful
0926 14:39:45.257658 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1156,"thread_start_us":106,"threads_started":1}
I20260926 14:39:45.259508 21227 tablet_service.cc:1467] Processing CreateTablet for tablet cc323f15ad654f41ac46fc4ec226e71b (DEFAULT_TABLE table=test-table [id=bd69955c01e2453f9751af6014d770f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:45.259727 20961 tablet_service.cc:1467] Processing CreateTablet for tablet cc323f15ad654f41ac46fc4ec226e71b (DEFAULT_TABLE table=test-table [id=bd69955c01e2453f9751af6014d770f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:45.260699 21227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc323f15ad654f41ac46fc4ec226e71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:45.260708 20961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc323f15ad654f41ac46fc4ec226e71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:45.260382 21094 tablet_service.cc:1467] Processing CreateTablet for tablet cc323f15ad654f41ac46fc4ec226e71b (DEFAULT_TABLE table=test-table [id=bd69955c01e2453f9751af6014d770f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:45.261246 21094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc323f15ad654f41ac46fc4ec226e71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:45.266345 21310 tablet_bootstrap.cc:492] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f: Bootstrap starting.
I20260926 14:39:45.267624 21312 tablet_bootstrap.cc:492] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb: Bootstrap starting.
I20260926 14:39:45.269477 21310 tablet_bootstrap.cc:654] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:45.269793 21312 tablet_bootstrap.cc:654] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:45.270416 21310 log.cc:824] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:45.270846 21312 log.cc:824] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:45.271124 21311 tablet_bootstrap.cc:492] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51: Bootstrap starting.
I20260926 14:39:45.272476 21310 tablet_bootstrap.cc:492] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f: No bootstrap required, opened a new log
I20260926 14:39:45.272666 21310 ts_tablet_manager.cc:1397] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:39:45.272769 21312 tablet_bootstrap.cc:492] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb: No bootstrap required, opened a new log
I20260926 14:39:45.272941 21312 ts_tablet_manager.cc:1397] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:39:45.273217 21311 tablet_bootstrap.cc:654] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:45.274266 21311 log.cc:824] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:45.276081 21310 raft_consensus.cc:348] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dd67c2f6e0474da6fb658351e75f51" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34511 } } peers { permanent_uuid: "da605c2b00cb439cbbbffa67c3a94acb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 32875 } } peers { permanent_uuid: "efdcdf2da82845eba2550d93811e817f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 44605 } }
I20260926 14:39:45.276338 21311 tablet_bootstrap.cc:492] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51: No bootstrap required, opened a new log
I20260926 14:39:45.276343 21310 raft_consensus.cc:374] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:45.276396 21310 raft_consensus.cc:729] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdcdf2da82845eba2550d93811e817f, State: Initialized, Role: FOLLOWER
I20260926 14:39:45.276503 21311 ts_tablet_manager.cc:1397] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260926 14:39:45.276553 21312 raft_consensus.cc:348] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dd67c2f6e0474da6fb658351e75f51" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34511 } } peers { permanent_uuid: "da605c2b00cb439cbbbffa67c3a94acb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 32875 } } peers { permanent_uuid: "efdcdf2da82845eba2550d93811e817f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 44605 } }
I20260926 14:39:45.276815 21312 raft_consensus.cc:374] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:45.276872 21312 raft_consensus.cc:729] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da605c2b00cb439cbbbffa67c3a94acb, State: Initialized, Role: FOLLOWER
I20260926 14:39:45.276818 21310 consensus_queue.cc:260] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dd67c2f6e0474da6fb658351e75f51" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34511 } } peers { permanent_uuid: "da605c2b00cb439cbbbffa67c3a94acb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 32875 } } peers { permanent_uuid: "efdcdf2da82845eba2550d93811e817f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 44605 } }
I20260926 14:39:45.277266 21312 consensus_queue.cc:260] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dd67c2f6e0474da6fb658351e75f51" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34511 } } peers { permanent_uuid: "da605c2b00cb439cbbbffa67c3a94acb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 32875 } } peers { permanent_uuid: "efdcdf2da82845eba2550d93811e817f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 44605 } }
I20260926 14:39:45.277701 21310 ts_tablet_manager.cc:1428] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:39:45.277951 21293 heartbeater.cc:503] Master 127.16.90.254:40689 was elected leader, sending a full tablet report...
I20260926 14:39:45.277989 21027 heartbeater.cc:503] Master 127.16.90.254:40689 was elected leader, sending a full tablet report...
I20260926 14:39:45.278172 21312 ts_tablet_manager.cc:1428] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:39:45.281109 21311 raft_consensus.cc:348] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dd67c2f6e0474da6fb658351e75f51" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34511 } } peers { permanent_uuid: "da605c2b00cb439cbbbffa67c3a94acb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 32875 } } peers { permanent_uuid: "efdcdf2da82845eba2550d93811e817f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 44605 } }
I20260926 14:39:45.281410 21311 raft_consensus.cc:374] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:45.281483 21311 raft_consensus.cc:729] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84dd67c2f6e0474da6fb658351e75f51, State: Initialized, Role: FOLLOWER
I20260926 14:39:45.281919 21311 consensus_queue.cc:260] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dd67c2f6e0474da6fb658351e75f51" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34511 } } peers { permanent_uuid: "da605c2b00cb439cbbbffa67c3a94acb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 32875 } } peers { permanent_uuid: "efdcdf2da82845eba2550d93811e817f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 44605 } }
I20260926 14:39:45.282793 21160 heartbeater.cc:503] Master 127.16.90.254:40689 was elected leader, sending a full tablet report...
I20260926 14:39:45.282842 21311 ts_tablet_manager.cc:1428] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
W20260926 14:39:45.302778 21028 tablet.cc:2369] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:39:45.362551 21316 raft_consensus.cc:482] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:45.362746 21316 raft_consensus.cc:504] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dd67c2f6e0474da6fb658351e75f51" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34511 } } peers { permanent_uuid: "da605c2b00cb439cbbbffa67c3a94acb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 32875 } } peers { permanent_uuid: "efdcdf2da82845eba2550d93811e817f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 44605 } }
I20260926 14:39:45.364145 21316 leader_election.cc:288] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84dd67c2f6e0474da6fb658351e75f51 (127.16.90.194:34511), da605c2b00cb439cbbbffa67c3a94acb (127.16.90.195:32875)
I20260926 14:39:45.368142 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.364497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:42745 (local address 127.16.90.194:34511)
0926 14:39:45.364612 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:45.364615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:45.364739 (+ 124us) server_negotiation.cc:407] Connection header received
0926 14:39:45.364874 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:45.364878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:45.364927 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:45.365018 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:45.365297 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.365915 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.367105 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.367556 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.367659 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.367737 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:45.367943 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 14:39:45.368023 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:39:45.368139 21319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.364392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:34511 (local address 127.16.90.193:42745)
0926 14:39:45.364635 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:45.364649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:45.364774 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:45.365042 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:45.365044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:45.365062 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:45.365189 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.365196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.366022 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.366025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.366974 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.366983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.367695 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.367699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.367791 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.367852 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 14:39:45.367915 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:39:45.367983 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":65,"thread_start_us":88,"threads_started":1}
I20260926 14:39:45.368184 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.364846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:33663 (local address 127.16.90.195:32875)
0926 14:39:45.364984 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:45.364989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:45.365131 (+ 142us) server_negotiation.cc:407] Connection header received
0926 14:39:45.365248 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:45.365251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:45.365299 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:45.365354 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:45.365607 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.366135 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.367174 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.367568 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.367700 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.367748 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:45.368011 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 14:39:45.368082 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:39:45.368290 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.364756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:32875 (local address 127.16.90.193:33663)
0926 14:39:45.365049 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:45.365062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:45.365155 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:45.365383 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:45.365386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:45.365401 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:45.365512 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.365518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.366260 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.366262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.367062 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:45.367073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.367712 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.367715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:45.367848 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.367865 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:39:45.367990 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 14:39:45.368091 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":9,"thread_start_us":97,"threads_started":1}
I20260926 14:39:45.368763 21247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc323f15ad654f41ac46fc4ec226e71b" candidate_uuid: "efdcdf2da82845eba2550d93811e817f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da605c2b00cb439cbbbffa67c3a94acb" is_pre_election: true
I20260926 14:39:45.368762 21114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc323f15ad654f41ac46fc4ec226e71b" candidate_uuid: "efdcdf2da82845eba2550d93811e817f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84dd67c2f6e0474da6fb658351e75f51" is_pre_election: true
I20260926 14:39:45.369057 21247 raft_consensus.cc:2454] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdcdf2da82845eba2550d93811e817f in term 0.
I20260926 14:39:45.369058 21114 raft_consensus.cc:2454] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdcdf2da82845eba2550d93811e817f in term 0.
I20260926 14:39:45.369504 20914 leader_election.cc:302] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da605c2b00cb439cbbbffa67c3a94acb, efdcdf2da82845eba2550d93811e817f; no voters:
I20260926 14:39:45.369756 21316 raft_consensus.cc:2781] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:45.369848 21316 raft_consensus.cc:482] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:45.369897 21316 raft_consensus.cc:3037] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:45.371027 21316 raft_consensus.cc:504] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dd67c2f6e0474da6fb658351e75f51" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34511 } } peers { permanent_uuid: "da605c2b00cb439cbbbffa67c3a94acb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 32875 } } peers { permanent_uuid: "efdcdf2da82845eba2550d93811e817f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 44605 } }
I20260926 14:39:45.371337 21316 leader_election.cc:288] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [CANDIDATE]: Term 1 election: Requested vote from peers 84dd67c2f6e0474da6fb658351e75f51 (127.16.90.194:34511), da605c2b00cb439cbbbffa67c3a94acb (127.16.90.195:32875)
I20260926 14:39:45.371740 21114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc323f15ad654f41ac46fc4ec226e71b" candidate_uuid: "efdcdf2da82845eba2550d93811e817f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84dd67c2f6e0474da6fb658351e75f51"
I20260926 14:39:45.371829 21247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc323f15ad654f41ac46fc4ec226e71b" candidate_uuid: "efdcdf2da82845eba2550d93811e817f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da605c2b00cb439cbbbffa67c3a94acb"
I20260926 14:39:45.371867 21114 raft_consensus.cc:3037] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:45.371944 21247 raft_consensus.cc:3037] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:45.372946 21114 raft_consensus.cc:2454] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdcdf2da82845eba2550d93811e817f in term 1.
I20260926 14:39:45.373108 21247 raft_consensus.cc:2454] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdcdf2da82845eba2550d93811e817f in term 1.
I20260926 14:39:45.373242 20914 leader_election.cc:302] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84dd67c2f6e0474da6fb658351e75f51, efdcdf2da82845eba2550d93811e817f; no voters:
I20260926 14:39:45.373396 21316 raft_consensus.cc:2781] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:45.373618 21316 raft_consensus.cc:686] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 1 LEADER]: Becoming Leader. State: Replica: efdcdf2da82845eba2550d93811e817f, State: Running, Role: LEADER
I20260926 14:39:45.373961 21316 consensus_queue.cc:237] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dd67c2f6e0474da6fb658351e75f51" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34511 } } peers { permanent_uuid: "da605c2b00cb439cbbbffa67c3a94acb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 32875 } } peers { permanent_uuid: "efdcdf2da82845eba2550d93811e817f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 44605 } }
I20260926 14:39:45.377156 20836 catalog_manager.cc:5495] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f reported cstate change: term changed from 0 to 1, leader changed from <none> to efdcdf2da82845eba2550d93811e817f (127.16.90.193). New cstate: current_term: 1 leader_uuid: "efdcdf2da82845eba2550d93811e817f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dd67c2f6e0474da6fb658351e75f51" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da605c2b00cb439cbbbffa67c3a94acb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 32875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efdcdf2da82845eba2550d93811e817f" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 44605 } health_report { overall_health: HEALTHY } } }
I20260926 14:39:45.407819 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.404765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.16.90.193:44605)
0926 14:39:45.404882 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:45.404886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:45.404899 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:39:45.404991 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:45.404993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:45.405043 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:45.405126 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:45.405325 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.405817 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.406577 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:45.406781 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:45.406843 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:45.407066 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:45.407480 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:45.407565 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:45.407612 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:39:45.407658 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:39:45.413604 21114 raft_consensus.cc:1261] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51 [term 1 FOLLOWER]: Refusing update from remote peer efdcdf2da82845eba2550d93811e817f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:45.413657 21247 raft_consensus.cc:1261] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb [term 1 FOLLOWER]: Refusing update from remote peer efdcdf2da82845eba2550d93811e817f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:45.414275 21321 consensus_queue.cc:1035] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84dd67c2f6e0474da6fb658351e75f51" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:45.414451 21316 consensus_queue.cc:1035] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da605c2b00cb439cbbbffa67c3a94acb" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 32875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:45.420583 21324 mvcc.cc:205] Tried to move back new op lower bound from 7333615965847232512 to 7333615965694488576. Current Snapshot: MvccSnapshot[applied={T|T < 7333615965847232512}]
I20260926 14:39:45.422641 21326 mvcc.cc:205] Tried to move back new op lower bound from 7333615965847232512 to 7333615965694488576. Current Snapshot: MvccSnapshot[applied={T|T < 7333615965847232512}]
I20260926 14:39:45.422919 21325 mvcc.cc:205] Tried to move back new op lower bound from 7333615965847232512 to 7333615965694488576. Current Snapshot: MvccSnapshot[applied={T|T < 7333615965847232512}]
I20260926 14:39:45.429004 20837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:45.429198 20837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:45.431658 20837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf6d554d1890492e80ecce432a5f7454: Sending DeleteTablet for 3 replicas of tablet cc323f15ad654f41ac46fc4ec226e71b
I20260926 14:39:45.432420 21227 tablet_service.cc:1514] Processing DeleteTablet for tablet cc323f15ad654f41ac46fc4ec226e71b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:45 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:44486
I20260926 14:39:45.432550 20961 tablet_service.cc:1514] Processing DeleteTablet for tablet cc323f15ad654f41ac46fc4ec226e71b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:45 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:34764
I20260926 14:39:45.432627 21094 tablet_service.cc:1514] Processing DeleteTablet for tablet cc323f15ad654f41ac46fc4ec226e71b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:45 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:40714
I20260926 14:39:45.433022 21334 tablet_replica.cc:330] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f: stopping tablet replica
I20260926 14:39:45.433130 21335 tablet_replica.cc:330] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51: stopping tablet replica
I20260926 14:39:45.433336 21334 raft_consensus.cc:2229] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:39:45.433424 21335 raft_consensus.cc:2229] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:45.433650 21334 raft_consensus.cc:2258] T cc323f15ad654f41ac46fc4ec226e71b P efdcdf2da82845eba2550d93811e817f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:45.433724 21335 raft_consensus.cc:2258] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:45.434073 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 20899
I20260926 14:39:45.435506 21335 ts_tablet_manager.cc:1905] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:45.436318 21333 tablet_replica.cc:330] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb: stopping tablet replica
I20260926 14:39:45.436622 21333 raft_consensus.cc:2229] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:45.436923 21333 raft_consensus.cc:2258] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:45.438199 21335 ts_tablet_manager.cc:1918] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:39:45.438297 21335 log.cc:1196] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/cc323f15ad654f41ac46fc4ec226e71b
I20260926 14:39:45.438637 21335 ts_tablet_manager.cc:1939] T cc323f15ad654f41ac46fc4ec226e71b P 84dd67c2f6e0474da6fb658351e75f51: Deleting consensus metadata
I20260926 14:39:45.439263 21333 ts_tablet_manager.cc:1905] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:45.439867 20821 catalog_manager.cc:4841] TS 84dd67c2f6e0474da6fb658351e75f51 (127.16.90.194:34511): tablet cc323f15ad654f41ac46fc4ec226e71b (table test-table [id=bd69955c01e2453f9751af6014d770f4]) successfully deleted
I20260926 14:39:45.441319 21333 ts_tablet_manager.cc:1918] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:39:45.441401 21333 log.cc:1196] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/cc323f15ad654f41ac46fc4ec226e71b
I20260926 14:39:45.441641 21333 ts_tablet_manager.cc:1939] T cc323f15ad654f41ac46fc4ec226e71b P da605c2b00cb439cbbbffa67c3a94acb: Deleting consensus metadata
I20260926 14:39:45.442648 20821 catalog_manager.cc:4841] TS da605c2b00cb439cbbbffa67c3a94acb (127.16.90.195:32875): tablet cc323f15ad654f41ac46fc4ec226e71b (table test-table [id=bd69955c01e2453f9751af6014d770f4]) successfully deleted
W20260926 14:39:45.442761 20824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:44605 (error 108)
I20260926 14:39:45.442894 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 21032
I20260926 14:39:45.444370 21306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.444030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:44605 (local address 127.0.0.1:34782)
0926 14:39:45.444212 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:45.444281 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:44605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260926 14:39:45.444552 20824 catalog_manager.cc:4584] TS efdcdf2da82845eba2550d93811e817f (127.16.90.193:44605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc323f15ad654f41ac46fc4ec226e71b: Network error: Client connection negotiation failed: client connection to 127.16.90.193:44605: connect: Connection refused (error 111)
I20260926 14:39:45.447791 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 21165
I20260926 14:39:45.452510 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 20806
2026-09-26T14:39:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2164 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T14:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:45Z Disabled control of system clock
I20260926 14:39:45.479885 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:44029
--webserver_interface=127.16.90.254
--webserver_port=0
--builtin_ntp_servers=127.16.90.212:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:44029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 14:39:45.583313 21340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:45.583557 21340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:45.583604 21340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:45.587468 21340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:45.587523 21340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:45.587538 21340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:45.587553 21340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:45.587565 21340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:45.592082 21340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:44029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:44029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:45.593055 21340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:45.594363 21340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:45.602103 21345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:45.602161 21346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:45.602170 21348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:45.602387 21340 server_base.cc:1034] running on GCE node
I20260926 14:39:45.602972 21340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:45.604058 21340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:45.605211 21340 hybrid_clock.cc:648] HybridClock initialized: now 1790433585605199 us; error 24 us; skew 500 ppm
I20260926 14:39:45.606892 21340 webserver.cc:460] Webserver started at http://127.16.90.254:46017/ using document root <none> and password file <none>
I20260926 14:39:45.607452 21340 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:45.607522 21340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:45.607772 21340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:45.609723 21340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "6fc1ab5eb0e74f90afd9b12ac7b15b02"
format_stamp: "Formatted at 2026-09-26 14:39:45 on dist-test-slave-x4sn"
server_key: "87f257f4e3a717d47eaa6c86b498b856"
server_key_iv: "deec15a311a9c37a1f1aad133b45eb57"
server_key_version: "encryptionkey@0"
I20260926 14:39:45.610245 21340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "6fc1ab5eb0e74f90afd9b12ac7b15b02"
format_stamp: "Formatted at 2026-09-26 14:39:45 on dist-test-slave-x4sn"
server_key: "87f257f4e3a717d47eaa6c86b498b856"
server_key_iv: "deec15a311a9c37a1f1aad133b45eb57"
server_key_version: "encryptionkey@0"
I20260926 14:39:45.613729 21340 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:39:45.616047 21354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:45.617273 21340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:45.617390 21340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "6fc1ab5eb0e74f90afd9b12ac7b15b02"
format_stamp: "Formatted at 2026-09-26 14:39:45 on dist-test-slave-x4sn"
server_key: "87f257f4e3a717d47eaa6c86b498b856"
server_key_iv: "deec15a311a9c37a1f1aad133b45eb57"
server_key_version: "encryptionkey@0"
I20260926 14:39:45.617486 21340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:45.621903 21340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:45.622466 21340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:45.622615 21340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:45.628842 21340 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:44029
I20260926 14:39:45.628876 21406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:44029 every 8 connection(s)
I20260926 14:39:45.629715 21340 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:45.632741 21407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:45.634469 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 21340
I20260926 14:39:45.634569 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:45.634846 16747 external_mini_cluster.cc:1428] Setting key add87ddec98d3dfe548046ac9eb2927c
I20260926 14:39:45.639111 21407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02: Bootstrap starting.
I20260926 14:39:45.640234 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.636268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.16.90.254:44029)
0926 14:39:45.636888 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:45.636903 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:45.636938 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:39:45.637598 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:45.637615 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:45.637953 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:45.638262 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:45.638993 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:45.638997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:45.639009 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:45.639020 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:45.639073 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:45.639089 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:45.639221 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:45.639291 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 14:39:45.639525 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":126,"threads_started":1}
I20260926 14:39:45.641296 21407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:45.641980 21407 log.cc:824] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:45.643751 21407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02: No bootstrap required, opened a new log
I20260926 14:39:45.647153 21407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc1ab5eb0e74f90afd9b12ac7b15b02" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44029 } }
I20260926 14:39:45.647354 21407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:45.647403 21407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc1ab5eb0e74f90afd9b12ac7b15b02, State: Initialized, Role: FOLLOWER
I20260926 14:39:45.647797 21407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc1ab5eb0e74f90afd9b12ac7b15b02" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44029 } }
I20260926 14:39:45.647918 21407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:45.647977 21407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:45.648063 21407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:45.648859 21407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc1ab5eb0e74f90afd9b12ac7b15b02" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44029 } }
I20260926 14:39:45.649158 21407 leader_election.cc:302] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fc1ab5eb0e74f90afd9b12ac7b15b02; no voters:
I20260926 14:39:45.649421 21407 leader_election.cc:288] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:45.649501 21412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:45.649662 21412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [term 1 LEADER]: Becoming Leader. State: Replica: 6fc1ab5eb0e74f90afd9b12ac7b15b02, State: Running, Role: LEADER
I20260926 14:39:45.649938 21412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc1ab5eb0e74f90afd9b12ac7b15b02" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44029 } }
I20260926 14:39:45.650607 21407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:45.651527 21414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc1ab5eb0e74f90afd9b12ac7b15b02. Latest consensus state: current_term: 1 leader_uuid: "6fc1ab5eb0e74f90afd9b12ac7b15b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc1ab5eb0e74f90afd9b12ac7b15b02" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44029 } } }
I20260926 14:39:45.651646 21414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:45.651953 21413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fc1ab5eb0e74f90afd9b12ac7b15b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc1ab5eb0e74f90afd9b12ac7b15b02" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44029 } } }
I20260926 14:39:45.652029 21413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:45.652060 21421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:45.654762 21421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:45.659991 21421 catalog_manager.cc:1347] Generated new cluster ID: fa955ba1138a4dc298164b172568f359
I20260926 14:39:45.660056 21421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:45.682987 21421 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:45.683338 21421 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:45.690646 21421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02: Generated new TSK 0
I20260926 14:39:45.691286 21421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:39:45.705467 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44029
--builtin_ntp_servers=127.16.90.212:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 14:39:45.814678 21431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:45.814977 21431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:45.815037 21431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:45.819265 21431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:45.819330 21431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:45.819408 21431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:45.824214 21431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:45.825271 21431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:45.826606 21431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:45.835711 21439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:45.835788 21436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:45.835829 21431 server_base.cc:1034] running on GCE node
W20260926 14:39:45.835788 21437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:45.836480 21431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:45.837069 21431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:45.838238 21431 hybrid_clock.cc:648] HybridClock initialized: now 1790433585838224 us; error 25 us; skew 500 ppm
I20260926 14:39:45.840080 21431 webserver.cc:460] Webserver started at http://127.16.90.193:41411/ using document root <none> and password file <none>
I20260926 14:39:45.840626 21431 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:45.840695 21431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:45.840906 21431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:45.842942 21431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "1cf0e282e9e648c281e7c83b63e9ecaa"
format_stamp: "Formatted at 2026-09-26 14:39:45 on dist-test-slave-x4sn"
server_key: "332581a6d71445ca96e348db9b0e6668"
server_key_iv: "d25e9aee9d8aef07e9dfb98c1722a03a"
server_key_version: "encryptionkey@0"
I20260926 14:39:45.843494 21431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "1cf0e282e9e648c281e7c83b63e9ecaa"
format_stamp: "Formatted at 2026-09-26 14:39:45 on dist-test-slave-x4sn"
server_key: "332581a6d71445ca96e348db9b0e6668"
server_key_iv: "d25e9aee9d8aef07e9dfb98c1722a03a"
server_key_version: "encryptionkey@0"
I20260926 14:39:45.847167 21431 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:39:45.849646 21445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:45.850953 21431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:45.851051 21431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "1cf0e282e9e648c281e7c83b63e9ecaa"
format_stamp: "Formatted at 2026-09-26 14:39:45 on dist-test-slave-x4sn"
server_key: "332581a6d71445ca96e348db9b0e6668"
server_key_iv: "d25e9aee9d8aef07e9dfb98c1722a03a"
server_key_version: "encryptionkey@0"
I20260926 14:39:45.851142 21431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:45.855490 21431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:45.856070 21431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:45.856217 21431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:45.856709 21431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:45.857522 21431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:45.857574 21431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:45.857622 21431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:45.857649 21431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:45.865499 21431 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:42547
I20260926 14:39:45.865520 21558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:42547 every 8 connection(s)
I20260926 14:39:45.866302 21431 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:39:45.871387 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 21431
I20260926 14:39:45.871520 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:39:45.871886 16747 external_mini_cluster.cc:1428] Setting key 190fab8cfd3e6fe0bcc962f1b1244c42
I20260926 14:39:45.872229 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.867952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:35253 (local address 127.16.90.254:44029)
0926 14:39:45.868074 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:45.868077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:45.869092 (+ 1015us) server_negotiation.cc:407] Connection header received
0926 14:39:45.870371 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:45.870375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:45.870419 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:45.870491 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:45.871627 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:45.871631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:45.871633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:45.871648 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:45.871713 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:45.871725 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:45.871793 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:45.872051 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 14:39:45.872123 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:39:45.873169 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:45.868331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44029 (local address 127.16.90.193:35253)
0926 14:39:45.868952 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:45.868994 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:45.870160 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:45.870657 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:45.870664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:45.870752 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:45.871306 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:39:45.871308 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:45.871361 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:39:45.871448 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:45.871863 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:45.871870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:45.871875 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:39:45.872040 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 14:39:45.872304 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260926 14:39:45.873833 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44029
--builtin_ntp_servers=127.16.90.212:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 14:39:45.874555 21559 heartbeater.cc:347] Connected to a master server at 127.16.90.254:44029
I20260926 14:39:45.874765 21559 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:45.875219 21559 heartbeater.cc:511] Master 127.16.90.254:44029 requested a full tablet report, sending...
I20260926 14:39:45.876688 21371 ts_manager.cc:194] Registered new tserver with Master: 1cf0e282e9e648c281e7c83b63e9ecaa (127.16.90.193:42547)
W20260926 14:39:45.980331 21562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:45.980592 21562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:45.980647 21562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:45.984678 21562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:45.984731 21562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:45.984808 21562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:45.989516 21562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:45.990553 21562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:45.991992 21562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:46.001346 21570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:46.001341 21568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:46.001341 21567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:46.001576 21562 server_base.cc:1034] running on GCE node
I20260926 14:39:46.002070 21562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:46.002689 21562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:46.003839 21562 hybrid_clock.cc:648] HybridClock initialized: now 1790433586003817 us; error 36 us; skew 500 ppm
I20260926 14:39:46.005527 21562 webserver.cc:460] Webserver started at http://127.16.90.194:42595/ using document root <none> and password file <none>
I20260926 14:39:46.006075 21562 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:46.006143 21562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:46.006335 21562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:46.008405 21562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "e4620f55e9ab47afb8827774d089f55f"
format_stamp: "Formatted at 2026-09-26 14:39:46 on dist-test-slave-x4sn"
server_key: "387bb85d45732bd37e9ea17aa3650831"
server_key_iv: "72313e0e583e911e54e13b5323b52d19"
server_key_version: "encryptionkey@0"
I20260926 14:39:46.008934 21562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "e4620f55e9ab47afb8827774d089f55f"
format_stamp: "Formatted at 2026-09-26 14:39:46 on dist-test-slave-x4sn"
server_key: "387bb85d45732bd37e9ea17aa3650831"
server_key_iv: "72313e0e583e911e54e13b5323b52d19"
server_key_version: "encryptionkey@0"
I20260926 14:39:46.012559 21562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:46.015609 21576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:46.016896 21562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:39:46.017004 21562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "e4620f55e9ab47afb8827774d089f55f"
format_stamp: "Formatted at 2026-09-26 14:39:46 on dist-test-slave-x4sn"
server_key: "387bb85d45732bd37e9ea17aa3650831"
server_key_iv: "72313e0e583e911e54e13b5323b52d19"
server_key_version: "encryptionkey@0"
I20260926 14:39:46.017107 21562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:46.021688 21562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:46.022293 21562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:46.022441 21562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:46.022948 21562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:46.023857 21562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:46.023904 21562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:46.023954 21562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:46.023980 21562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:46.032318 21562 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:33843
I20260926 14:39:46.032356 21689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:33843 every 8 connection(s)
I20260926 14:39:46.033169 21562 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:39:46.038705 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.034838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:40727 (local address 127.16.90.254:44029)
0926 14:39:46.034971 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:46.034975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:46.036062 (+ 1087us) server_negotiation.cc:407] Connection header received
0926 14:39:46.037291 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:46.037293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:46.037329 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:46.037402 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:46.038349 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:46.038352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:46.038353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:46.038365 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:46.038384 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:46.038392 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:46.038465 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:46.038547 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 14:39:46.038582 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:39:46.039532 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.035248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44029 (local address 127.16.90.194:40727)
0926 14:39:46.035923 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:46.035963 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:46.037107 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:46.037546 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:46.037553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:46.037642 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:46.038118 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:39:46.038120 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:46.038158 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:39:46.038221 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:46.038453 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:46.038457 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:46.038461 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:39:46.038581 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 14:39:46.038768 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260926 14:39:46.039736 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 21562
I20260926 14:39:46.039857 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:39:46.040110 16747 external_mini_cluster.cc:1428] Setting key 125192776f5901f954b48b50894f221b
I20260926 14:39:46.040704 21690 heartbeater.cc:347] Connected to a master server at 127.16.90.254:44029
I20260926 14:39:46.040921 21690 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:46.041311 21690 heartbeater.cc:511] Master 127.16.90.254:44029 requested a full tablet report, sending...
I20260926 14:39:46.042101 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44029
--builtin_ntp_servers=127.16.90.212:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 14:39:46.042480 21371 ts_manager.cc:194] Registered new tserver with Master: e4620f55e9ab47afb8827774d089f55f (127.16.90.194:33843)
W20260926 14:39:46.149991 21693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:46.150251 21693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:46.150288 21693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:46.154342 21693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:46.154394 21693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:46.154448 21693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:46.159080 21693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:46.160108 21693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:46.161403 21693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:46.170747 21701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:46.170849 21699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:46.170912 21698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:46.171226 21693 server_base.cc:1034] running on GCE node
I20260926 14:39:46.171649 21693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:46.172313 21693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:46.173480 21693 hybrid_clock.cc:648] HybridClock initialized: now 1790433586173469 us; error 26 us; skew 500 ppm
I20260926 14:39:46.175204 21693 webserver.cc:460] Webserver started at http://127.16.90.195:45277/ using document root <none> and password file <none>
I20260926 14:39:46.175765 21693 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:46.175828 21693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:46.175992 21693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:46.177932 21693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "cd983ffcd8604af98b3f2edb7c1c50b8"
format_stamp: "Formatted at 2026-09-26 14:39:46 on dist-test-slave-x4sn"
server_key: "6f194ee64e8c05ab7bf14749a48207e4"
server_key_iv: "73b6caaa3a8e26f3d0fde63b8c84b9ab"
server_key_version: "encryptionkey@0"
I20260926 14:39:46.178406 21693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "cd983ffcd8604af98b3f2edb7c1c50b8"
format_stamp: "Formatted at 2026-09-26 14:39:46 on dist-test-slave-x4sn"
server_key: "6f194ee64e8c05ab7bf14749a48207e4"
server_key_iv: "73b6caaa3a8e26f3d0fde63b8c84b9ab"
server_key_version: "encryptionkey@0"
I20260926 14:39:46.182055 21693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:39:46.184389 21707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:46.185550 21693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:46.185647 21693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "cd983ffcd8604af98b3f2edb7c1c50b8"
format_stamp: "Formatted at 2026-09-26 14:39:46 on dist-test-slave-x4sn"
server_key: "6f194ee64e8c05ab7bf14749a48207e4"
server_key_iv: "73b6caaa3a8e26f3d0fde63b8c84b9ab"
server_key_version: "encryptionkey@0"
I20260926 14:39:46.185756 21693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:46.190068 21693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:46.190688 21693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:46.190856 21693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:46.191360 21693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:46.192323 21693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:46.192369 21693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:46.192418 21693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:46.192444 21693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:46.200901 21693 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:33009
I20260926 14:39:46.200925 21820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:33009 every 8 connection(s)
I20260926 14:39:46.201786 21693 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:46.207624 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.203500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:57711 (local address 127.16.90.254:44029)
0926 14:39:46.203627 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:46.203630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:46.204732 (+ 1102us) server_negotiation.cc:407] Connection header received
0926 14:39:46.206032 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:46.206036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:46.206082 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:46.206158 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:46.207167 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:46.207172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:46.207174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:46.207187 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:46.207210 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:46.207219 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:46.207299 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:46.207419 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 14:39:46.207463 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:39:46.208243 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 21693
I20260926 14:39:46.208352 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:39:46.208509 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.203933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44029 (local address 127.16.90.195:57711)
0926 14:39:46.204586 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:46.204625 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:46.205823 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:46.206333 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:46.206340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:46.206431 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:46.206921 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:39:46.206923 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:46.206962 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:39:46.207028 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:46.207289 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:46.207297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:46.207303 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:39:46.207457 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 14:39:46.207730 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":142,"threads_started":1}
I20260926 14:39:46.208653 16747 external_mini_cluster.cc:1428] Setting key 453364cc64a62f8151db6d638ea82dce
I20260926 14:39:46.209673 21821 heartbeater.cc:347] Connected to a master server at 127.16.90.254:44029
I20260926 14:39:46.209909 21821 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:46.210331 21821 heartbeater.cc:511] Master 127.16.90.254:44029 requested a full tablet report, sending...
I20260926 14:39:46.211345 21370 ts_manager.cc:194] Registered new tserver with Master: cd983ffcd8604af98b3f2edb7c1c50b8 (127.16.90.195:33009)
I20260926 14:39:46.222620 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:39:46.225998 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.224696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.16.90.254:44029)
0926 14:39:46.224838 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:46.224842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:46.224904 (+ 62us) server_negotiation.cc:407] Connection header received
0926 14:39:46.225007 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:46.225009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:46.225041 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:46.225100 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:46.225713 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:46.225715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:46.225717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:46.225724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:46.225738 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:46.225744 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:46.225799 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:46.225839 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 14:39:46.225868 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 14:39:46.228674 21370 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:39:46.231612 21370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:46.242856 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.240716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.16.90.195:33009)
0926 14:39:46.241132 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:46.241144 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:46.241165 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:39:46.241244 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:46.241250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:46.241423 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:46.241550 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:46.242304 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:46.242311 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:46.242324 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:46.242336 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:46.242378 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:46.242406 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:46.242496 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:46.242638 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 14:39:46.242721 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":117,"threads_started":1}
I20260926 14:39:46.242846 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.239594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:33843 (local address 127.0.0.1:39320)
0926 14:39:46.240390 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:46.240438 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:46.240532 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:46.241329 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:46.241335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:46.241407 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:46.241945 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:39:46.241947 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:46.241967 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:39:46.242034 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:46.242561 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:46.242565 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:46.242568 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:39:46.242623 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 14:39:46.242688 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":86,"threads_started":1}
I20260926 14:39:46.242846 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.240289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33009 (local address 127.0.0.1:33470)
0926 14:39:46.240600 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:46.240622 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:46.240699 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:46.241671 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:46.241674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:46.241699 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:46.242162 (+ 463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:39:46.242163 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:46.242172 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:39:46.242193 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:46.242481 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:46.242490 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:46.242503 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 14:39:46.242620 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 14:39:46.242692 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":93,"threads_started":1}
I20260926 14:39:46.242882 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.239896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.16.90.193:42547)
0926 14:39:46.240215 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:46.240227 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:46.240634 (+ 407us) server_negotiation.cc:407] Connection header received
0926 14:39:46.240823 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:46.240829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:46.241028 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:46.241133 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:46.242371 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:46.242375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:46.242385 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:46.242395 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:46.242433 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:46.242461 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:46.242555 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:46.242684 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 14:39:46.242754 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":117,"threads_started":1}
I20260926 14:39:46.243424 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.240285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.16.90.194:33843)
0926 14:39:46.240926 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:46.240937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:46.240958 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:39:46.241014 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:46.241019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:46.241155 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:46.241311 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:46.242368 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:46.242374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:46.242385 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:46.242395 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:46.242433 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:46.242457 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:46.243170 (+ 713us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:46.243227 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 14:39:46.243286 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":93,"threads_started":1}
I20260926 14:39:46.243429 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.239636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:42547 (local address 127.0.0.1:51060)
0926 14:39:46.240531 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:46.240546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:46.240699 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:46.241845 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:46.241848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:46.241863 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:46.242206 (+ 343us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:39:46.242208 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:46.242219 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:39:46.242245 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:46.242548 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:46.242551 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:46.242554 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:39:46.243293 (+ 739us) client_negotiation.cc:240] Negotiation successful
0926 14:39:46.243326 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":118,"threads_started":1}
I20260926 14:39:46.245647 21755 tablet_service.cc:1467] Processing CreateTablet for tablet 171739e382754cf9bec625f8d3544727 (DEFAULT_TABLE table=test-table [id=ccb8765ee92f49ae91f89d6aa59b7e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:46.245954 21624 tablet_service.cc:1467] Processing CreateTablet for tablet 171739e382754cf9bec625f8d3544727 (DEFAULT_TABLE table=test-table [id=ccb8765ee92f49ae91f89d6aa59b7e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:46.246006 21493 tablet_service.cc:1467] Processing CreateTablet for tablet 171739e382754cf9bec625f8d3544727 (DEFAULT_TABLE table=test-table [id=ccb8765ee92f49ae91f89d6aa59b7e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:46.246560 21755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171739e382754cf9bec625f8d3544727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:46.246955 21624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171739e382754cf9bec625f8d3544727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:46.246955 21493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171739e382754cf9bec625f8d3544727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:46.252455 21838 tablet_bootstrap.cc:492] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8: Bootstrap starting.
I20260926 14:39:46.254136 21839 tablet_bootstrap.cc:492] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f: Bootstrap starting.
I20260926 14:39:46.254136 21840 tablet_bootstrap.cc:492] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa: Bootstrap starting.
I20260926 14:39:46.254752 21838 tablet_bootstrap.cc:654] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:46.255654 21838 log.cc:824] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:46.256210 21839 tablet_bootstrap.cc:654] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:46.256213 21840 tablet_bootstrap.cc:654] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:46.257015 21840 log.cc:824] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:46.257015 21839 log.cc:824] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:46.257335 21838 tablet_bootstrap.cc:492] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8: No bootstrap required, opened a new log
I20260926 14:39:46.257529 21838 ts_tablet_manager.cc:1397] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260926 14:39:46.258545 21839 tablet_bootstrap.cc:492] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f: No bootstrap required, opened a new log
I20260926 14:39:46.258709 21839 ts_tablet_manager.cc:1397] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:39:46.258806 21840 tablet_bootstrap.cc:492] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa: No bootstrap required, opened a new log
I20260926 14:39:46.258966 21840 ts_tablet_manager.cc:1397] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:39:46.261842 21839 raft_consensus.cc:348] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4620f55e9ab47afb8827774d089f55f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33843 } } peers { permanent_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42547 } } peers { permanent_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33009 } }
I20260926 14:39:46.262045 21839 raft_consensus.cc:374] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:46.262094 21839 raft_consensus.cc:729] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4620f55e9ab47afb8827774d089f55f, State: Initialized, Role: FOLLOWER
I20260926 14:39:46.262475 21839 consensus_queue.cc:260] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4620f55e9ab47afb8827774d089f55f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33843 } } peers { permanent_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42547 } } peers { permanent_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33009 } }
I20260926 14:39:46.262562 21838 raft_consensus.cc:348] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4620f55e9ab47afb8827774d089f55f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33843 } } peers { permanent_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42547 } } peers { permanent_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33009 } }
I20260926 14:39:46.262804 21838 raft_consensus.cc:374] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:46.262866 21838 raft_consensus.cc:729] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd983ffcd8604af98b3f2edb7c1c50b8, State: Initialized, Role: FOLLOWER
I20260926 14:39:46.263161 21690 heartbeater.cc:503] Master 127.16.90.254:44029 was elected leader, sending a full tablet report...
I20260926 14:39:46.262981 21840 raft_consensus.cc:348] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4620f55e9ab47afb8827774d089f55f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33843 } } peers { permanent_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42547 } } peers { permanent_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33009 } }
I20260926 14:39:46.263227 21840 raft_consensus.cc:374] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:46.263276 21840 raft_consensus.cc:729] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf0e282e9e648c281e7c83b63e9ecaa, State: Initialized, Role: FOLLOWER
I20260926 14:39:46.263304 21838 consensus_queue.cc:260] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4620f55e9ab47afb8827774d089f55f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33843 } } peers { permanent_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42547 } } peers { permanent_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33009 } }
I20260926 14:39:46.263636 21840 consensus_queue.cc:260] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4620f55e9ab47afb8827774d089f55f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33843 } } peers { permanent_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42547 } } peers { permanent_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33009 } }
I20260926 14:39:46.263965 21821 heartbeater.cc:503] Master 127.16.90.254:44029 was elected leader, sending a full tablet report...
I20260926 14:39:46.264266 21838 ts_tablet_manager.cc:1428] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260926 14:39:46.264477 21840 ts_tablet_manager.cc:1428] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:39:46.264756 21559 heartbeater.cc:503] Master 127.16.90.254:44029 was elected leader, sending a full tablet report...
I20260926 14:39:46.265091 21839 ts_tablet_manager.cc:1428] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260926 14:39:46.284013 21691 tablet.cc:2369] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:39:46.367395 21560 tablet.cc:2369] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:39:46.385680 21846 raft_consensus.cc:482] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:46.385886 21846 raft_consensus.cc:504] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4620f55e9ab47afb8827774d089f55f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33843 } } peers { permanent_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42547 } } peers { permanent_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33009 } }
I20260926 14:39:46.386997 21846 leader_election.cc:288] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4620f55e9ab47afb8827774d089f55f (127.16.90.194:33843), cd983ffcd8604af98b3f2edb7c1c50b8 (127.16.90.195:33009)
I20260926 14:39:46.389021 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.387490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:36569 (local address 127.16.90.194:33843)
0926 14:39:46.387621 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:46.387625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:46.387685 (+ 60us) server_negotiation.cc:407] Connection header received
0926 14:39:46.387823 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:46.387826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:46.387863 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:46.387937 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:46.388616 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:46.388620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:46.388622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:46.388633 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:46.388652 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:46.388660 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:46.388731 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:46.388850 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 14:39:46.388894 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:39:46.389024 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.387571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:42549 (local address 127.16.90.195:33009)
0926 14:39:46.387716 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:46.387720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:46.387883 (+ 163us) server_negotiation.cc:407] Connection header received
0926 14:39:46.388007 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:46.388010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:46.388047 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:46.388095 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:46.388669 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:46.388672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:46.388675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:46.388700 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:46.388718 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:46.388728 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:46.388812 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:46.388888 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 14:39:46.388921 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:39:46.389029 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.387267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:33843 (local address 127.16.90.193:36569)
0926 14:39:46.387595 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:46.387612 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:46.387728 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:46.387958 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:46.387960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:46.387976 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:46.388459 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:39:46.388461 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:46.388472 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:39:46.388501 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:46.388748 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:46.388751 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:46.388754 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:39:46.388829 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 14:39:46.388864 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
I20260926 14:39:46.389050 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.387277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33009 (local address 127.16.90.193:42549)
0926 14:39:46.387790 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:46.387804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:46.387894 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:46.388225 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:46.388227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:46.388240 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:46.388546 (+ 306us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:39:46.388548 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:46.388557 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:39:46.388576 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:46.388800 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:46.388803 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:46.388806 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:39:46.388885 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 14:39:46.388915 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":92,"threads_started":1}
I20260926 14:39:46.389771 21775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171739e382754cf9bec625f8d3544727" candidate_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8" is_pre_election: true
I20260926 14:39:46.389771 21644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171739e382754cf9bec625f8d3544727" candidate_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4620f55e9ab47afb8827774d089f55f" is_pre_election: true
I20260926 14:39:46.390069 21775 raft_consensus.cc:2454] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf0e282e9e648c281e7c83b63e9ecaa in term 0.
I20260926 14:39:46.390069 21644 raft_consensus.cc:2454] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf0e282e9e648c281e7c83b63e9ecaa in term 0.
I20260926 14:39:46.390509 21448 leader_election.cc:302] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf0e282e9e648c281e7c83b63e9ecaa, e4620f55e9ab47afb8827774d089f55f; no voters:
I20260926 14:39:46.390748 21846 raft_consensus.cc:2781] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:46.390820 21846 raft_consensus.cc:482] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:46.390869 21846 raft_consensus.cc:3037] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:46.391899 21846 raft_consensus.cc:504] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4620f55e9ab47afb8827774d089f55f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33843 } } peers { permanent_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42547 } } peers { permanent_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33009 } }
I20260926 14:39:46.392206 21846 leader_election.cc:288] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [CANDIDATE]: Term 1 election: Requested vote from peers e4620f55e9ab47afb8827774d089f55f (127.16.90.194:33843), cd983ffcd8604af98b3f2edb7c1c50b8 (127.16.90.195:33009)
I20260926 14:39:46.392551 21775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171739e382754cf9bec625f8d3544727" candidate_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8"
I20260926 14:39:46.392547 21644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171739e382754cf9bec625f8d3544727" candidate_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4620f55e9ab47afb8827774d089f55f"
I20260926 14:39:46.392660 21775 raft_consensus.cc:3037] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:46.392699 21644 raft_consensus.cc:3037] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:46.393625 21644 raft_consensus.cc:2454] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf0e282e9e648c281e7c83b63e9ecaa in term 1.
I20260926 14:39:46.393632 21775 raft_consensus.cc:2454] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf0e282e9e648c281e7c83b63e9ecaa in term 1.
I20260926 14:39:46.393885 21446 leader_election.cc:302] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cf0e282e9e648c281e7c83b63e9ecaa, cd983ffcd8604af98b3f2edb7c1c50b8, e4620f55e9ab47afb8827774d089f55f; no voters:
I20260926 14:39:46.394100 21846 raft_consensus.cc:2781] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:46.394297 21846 raft_consensus.cc:686] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [term 1 LEADER]: Becoming Leader. State: Replica: 1cf0e282e9e648c281e7c83b63e9ecaa, State: Running, Role: LEADER
I20260926 14:39:46.394585 21846 consensus_queue.cc:237] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4620f55e9ab47afb8827774d089f55f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33843 } } peers { permanent_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42547 } } peers { permanent_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33009 } }
I20260926 14:39:46.397656 21371 catalog_manager.cc:5495] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cf0e282e9e648c281e7c83b63e9ecaa (127.16.90.193). New cstate: current_term: 1 leader_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4620f55e9ab47afb8827774d089f55f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cf0e282e9e648c281e7c83b63e9ecaa" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33009 } health_report { overall_health: UNKNOWN } } }
I20260926 14:39:46.430424 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.428954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.16.90.193:42547)
0926 14:39:46.429081 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:46.429085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:46.429100 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:39:46.429255 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:46.429258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:46.429319 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:46.429392 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:46.430061 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:46.430064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:46.430067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:46.430078 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:46.430099 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:46.430107 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:46.430181 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:46.430256 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 14:39:46.430294 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:39:46.436053 21644 raft_consensus.cc:1261] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f [term 1 FOLLOWER]: Refusing update from remote peer 1cf0e282e9e648c281e7c83b63e9ecaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:46.436053 21775 raft_consensus.cc:1261] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8 [term 1 FOLLOWER]: Refusing update from remote peer 1cf0e282e9e648c281e7c83b63e9ecaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:46.436860 21849 consensus_queue.cc:1035] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4620f55e9ab47afb8827774d089f55f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:46.437070 21846 consensus_queue.cc:1035] T 171739e382754cf9bec625f8d3544727 P 1cf0e282e9e648c281e7c83b63e9ecaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd983ffcd8604af98b3f2edb7c1c50b8" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:46.444123 21852 mvcc.cc:205] Tried to move back new op lower bound from 7333615970034642944 to 7333615969874427904. Current Snapshot: MvccSnapshot[applied={T|T < 7333615970034642944 or (T in {7333615970034642944})}]
I20260926 14:39:46.444554 21854 mvcc.cc:205] Tried to move back new op lower bound from 7333615970034642944 to 7333615969874427904. Current Snapshot: MvccSnapshot[applied={T|T < 7333615970034642944}]
I20260926 14:39:46.451761 21370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:46.451901 21370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:46.454174 21370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fc1ab5eb0e74f90afd9b12ac7b15b02: Sending DeleteTablet for 3 replicas of tablet 171739e382754cf9bec625f8d3544727
I20260926 14:39:46.454869 21624 tablet_service.cc:1514] Processing DeleteTablet for tablet 171739e382754cf9bec625f8d3544727 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:46 UTC) from {username='slave'} at 127.0.0.1:39320
I20260926 14:39:46.454906 21755 tablet_service.cc:1514] Processing DeleteTablet for tablet 171739e382754cf9bec625f8d3544727 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:46 UTC) from {username='slave'} at 127.0.0.1:33470
I20260926 14:39:46.454931 21493 tablet_service.cc:1514] Processing DeleteTablet for tablet 171739e382754cf9bec625f8d3544727 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:46 UTC) from {username='slave'} at 127.0.0.1:51060
I20260926 14:39:46.455437 21863 tablet_replica.cc:330] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8: stopping tablet replica
I20260926 14:39:46.455801 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 21431
I20260926 14:39:46.455806 21863 raft_consensus.cc:2229] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:46.456468 21863 raft_consensus.cc:2258] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:46.456768 21862 tablet_replica.cc:330] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f: stopping tablet replica
I20260926 14:39:46.457038 21862 raft_consensus.cc:2229] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:46.457284 21862 raft_consensus.cc:2258] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:46.458473 21863 ts_tablet_manager.cc:1905] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:46.458607 21862 ts_tablet_manager.cc:1905] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:46.460549 21863 ts_tablet_manager.cc:1918] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:39:46.460630 21863 log.cc:1196] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/171739e382754cf9bec625f8d3544727
I20260926 14:39:46.460875 21863 ts_tablet_manager.cc:1939] T 171739e382754cf9bec625f8d3544727 P cd983ffcd8604af98b3f2edb7c1c50b8: Deleting consensus metadata
I20260926 14:39:46.461077 21862 ts_tablet_manager.cc:1918] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:39:46.461164 21862 log.cc:1196] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/171739e382754cf9bec625f8d3544727
I20260926 14:39:46.461427 21862 ts_tablet_manager.cc:1939] T 171739e382754cf9bec625f8d3544727 P e4620f55e9ab47afb8827774d089f55f: Deleting consensus metadata
I20260926 14:39:46.461918 21355 catalog_manager.cc:4841] TS cd983ffcd8604af98b3f2edb7c1c50b8 (127.16.90.195:33009): tablet 171739e382754cf9bec625f8d3544727 (table test-table [id=ccb8765ee92f49ae91f89d6aa59b7e15]) successfully deleted
I20260926 14:39:46.462466 21357 catalog_manager.cc:4841] TS e4620f55e9ab47afb8827774d089f55f (127.16.90.194:33843): tablet 171739e382754cf9bec625f8d3544727 (table test-table [id=ccb8765ee92f49ae91f89d6aa59b7e15]) successfully deleted
W20260926 14:39:46.463227 21355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:42547 (error 108)
I20260926 14:39:46.463924 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 21562
I20260926 14:39:46.464169 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.463908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:42547 (local address 127.0.0.1:51074)
0926 14:39:46.464022 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:46.464092 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:42547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 14:39:46.464618 21355 catalog_manager.cc:4584] TS 1cf0e282e9e648c281e7c83b63e9ecaa (127.16.90.193:42547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 171739e382754cf9bec625f8d3544727: Network error: Client connection negotiation failed: client connection to 127.16.90.193:42547: connect: Connection refused (error 111)
I20260926 14:39:46.468971 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 21693
I20260926 14:39:46.473349 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 21340
2026-09-26T14:39:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1020 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 14:39:46.493042 16747 mini_oidc.cc:152] Starting JWKS server
I20260926 14:39:46.494632 16747 webserver.cc:460] Webserver started at https://127.0.0.1:46187/ using document root <none> and password file <none>
I20260926 14:39:46.494776 16747 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 14:39:46.494987 16747 webserver.cc:460] Webserver started at http://127.0.0.1:41005/ using document root <none> and password file <none>
2026-09-26T14:39:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:46Z Disabled control of system clock
I20260926 14:39:46.503248 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:41935
--webserver_interface=127.16.90.254
--webserver_port=0
--builtin_ntp_servers=127.16.90.212:35335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:41935
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46187/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/testchainca.cert with env {}
W20260926 14:39:46.607898 21871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:46.608148 21871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:46.608186 21871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:46.612157 21871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:46.612208 21871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:46.612221 21871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:46.612236 21871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:46.612252 21871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:39:46.612280 21871 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:39:46.612304 21871 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46187/jwks.json
I20260926 14:39:46.617552 21871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:41935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46187/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:46.618520 21871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:46.619491 21871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:46.627463 21877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:46.627465 21879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:46.627513 21871 server_base.cc:1034] running on GCE node
W20260926 14:39:46.627465 21876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:46.628247 21871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:46.629244 21871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:46.630402 21871 hybrid_clock.cc:648] HybridClock initialized: now 1790433586630378 us; error 35 us; skew 500 ppm
I20260926 14:39:46.632128 21871 webserver.cc:460] Webserver started at http://127.16.90.254:41243/ using document root <none> and password file <none>
I20260926 14:39:46.632654 21871 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:46.632715 21871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:46.632897 21871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:46.634785 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "c097f8f96c4645db867c37b2a2f26973"
format_stamp: "Formatted at 2026-09-26 14:39:46 on dist-test-slave-x4sn"
server_key: "1fc5bd4e74859979ff74224c4168df24"
server_key_iv: "ed46bb4d84351cb9dd33dbc64f300208"
server_key_version: "encryptionkey@0"
I20260926 14:39:46.635290 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "c097f8f96c4645db867c37b2a2f26973"
format_stamp: "Formatted at 2026-09-26 14:39:46 on dist-test-slave-x4sn"
server_key: "1fc5bd4e74859979ff74224c4168df24"
server_key_iv: "ed46bb4d84351cb9dd33dbc64f300208"
server_key_version: "encryptionkey@0"
I20260926 14:39:46.638717 21871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:39:46.640937 21885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:46.642174 21871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:39:46.642290 21871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "c097f8f96c4645db867c37b2a2f26973"
format_stamp: "Formatted at 2026-09-26 14:39:46 on dist-test-slave-x4sn"
server_key: "1fc5bd4e74859979ff74224c4168df24"
server_key_iv: "ed46bb4d84351cb9dd33dbc64f300208"
server_key_version: "encryptionkey@0"
I20260926 14:39:46.642383 21871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:46.663365 21871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:46.664198 21871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:46.664393 21871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:46.671176 21871 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:41935
I20260926 14:39:46.671186 21937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:41935 every 8 connection(s)
I20260926 14:39:46.672111 21871 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:46.675385 21938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:46.678387 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 21871
I20260926 14:39:46.678500 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:46.678747 16747 external_mini_cluster.cc:1428] Setting key 35ef97645eafb353d55e08666b42f50e
I20260926 14:39:46.681214 21938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973: Bootstrap starting.
I20260926 14:39:46.683390 21938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:46.684162 21938 log.cc:824] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:46.685940 21938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973: No bootstrap required, opened a new log
I20260926 14:39:46.687615 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.680123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.16.90.254:41935)
0926 14:39:46.680778 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:46.680793 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:46.680829 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:39:46.681478 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:46.681496 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:46.681909 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:46.682225 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:46.682647 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:46.683637 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:46.684576 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:46.685040 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:46.685196 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:46.685952 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:46.685957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:46.685970 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:46.686382 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:46.686440 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:46.686460 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:46.686544 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:46.686671 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 14:39:46.686926 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":133,"threads_started":1}
I20260926 14:39:46.689699 21938 raft_consensus.cc:348] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c097f8f96c4645db867c37b2a2f26973" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41935 } }
I20260926 14:39:46.689942 21938 raft_consensus.cc:374] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:46.690004 21938 raft_consensus.cc:729] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c097f8f96c4645db867c37b2a2f26973, State: Initialized, Role: FOLLOWER
I20260926 14:39:46.690403 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:39:46.690438 21938 consensus_queue.cc:260] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c097f8f96c4645db867c37b2a2f26973" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41935 } }
I20260926 14:39:46.690582 21938 raft_consensus.cc:388] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:46.690644 21938 raft_consensus.cc:482] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:46.690742 21938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:46.691589 21938 raft_consensus.cc:504] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c097f8f96c4645db867c37b2a2f26973" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41935 } }
I20260926 14:39:46.691926 21938 leader_election.cc:302] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c097f8f96c4645db867c37b2a2f26973; no voters:
I20260926 14:39:46.692190 21938 leader_election.cc:288] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:46.692270 21943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:46.692473 21943 raft_consensus.cc:686] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [term 1 LEADER]: Becoming Leader. State: Replica: c097f8f96c4645db867c37b2a2f26973, State: Running, Role: LEADER
I20260926 14:39:46.692777 21943 consensus_queue.cc:237] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c097f8f96c4645db867c37b2a2f26973" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41935 } }
I20260926 14:39:46.693305 21938 sys_catalog.cc:564] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:46.694376 21945 sys_catalog.cc:455] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c097f8f96c4645db867c37b2a2f26973. Latest consensus state: current_term: 1 leader_uuid: "c097f8f96c4645db867c37b2a2f26973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c097f8f96c4645db867c37b2a2f26973" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41935 } } }
I20260926 14:39:46.694376 21944 sys_catalog.cc:455] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c097f8f96c4645db867c37b2a2f26973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c097f8f96c4645db867c37b2a2f26973" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41935 } } }
I20260926 14:39:46.694562 21944 sys_catalog.cc:458] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:46.694811 21945 sys_catalog.cc:458] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:46.695013 21952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:46.697631 21952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:46.703132 21952 catalog_manager.cc:1347] Generated new cluster ID: 17429c3cde1b446a9c40e7215a340513
I20260926 14:39:46.703228 21952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:46.719852 21952 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:46.720788 21952 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:46.724345 21952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c097f8f96c4645db867c37b2a2f26973: Generated new TSK 0
I20260926 14:39:46.725104 21952 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 14:39:46.754299 21968 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 14:39:46.754417 21968 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 14:39:46.754891 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.737752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.16.90.254:41935)
0926 14:39:46.737872 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:46.737876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:46.738050 (+ 174us) server_negotiation.cc:407] Connection header received
0926 14:39:46.738270 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:46.738272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:46.738317 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:46.738398 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:39:46.738663 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:46.739173 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:46.740398 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:46.740610 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:46.740684 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:46.741001 (+ 317us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:39:46.754554 (+ 13553us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 14:39:46.754783 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57516: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 14:39:46.754959 21941 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57516: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 14:39:46.756803 21967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:39:46.737622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:41935 (local address 127.0.0.1:57516)
0926 14:39:46.737969 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:46.737984 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:46.738115 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:46.738427 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:46.738431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:46.738440 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:39:46.738574 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:46.738581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:46.739271 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:46.739274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:46.740295 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:46.740307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:46.740705 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:46.740709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:46.740799 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:46.740850 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:39:46.754780 (+ 13930us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 14:39:46.755038 (+ 258us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.90.254:41935: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":119,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.90.254:41935: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 14:39:46.757606 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 21871
2026-09-26T14:39:46Z chronyd exiting
I20260926 14:39:46.794116 16747 test_util.cc:175] -----------------------------------------------
I20260926 14:39:46.794216 16747 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433562403030-16747-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 14:39:46.795185 16747 mini_oidc.cc:152] Starting JWKS server
I20260926 14:39:46.796073 16747 webserver.cc:460] Webserver started at https://127.0.0.1:42287/ using document root <none> and password file <none>
I20260926 14:39:46.796171 16747 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 14:39:46.796412 16747 webserver.cc:460] Webserver started at http://127.0.0.1:42135/ using document root <none> and password file <none>
2026-09-26T14:39:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:46Z Disabled control of system clock
I20260926 14:39:46.804545 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:36833
--webserver_interface=127.16.90.254
--webserver_port=0
--builtin_ntp_servers=127.16.90.212:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:36833
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42287/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/testchainca.cert with env {}
W20260926 14:39:46.908632 21974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:46.908885 21974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:46.908927 21974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:46.912920 21974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:46.912982 21974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:46.912997 21974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:46.913012 21974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:46.913025 21974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:39:46.913045 21974 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:39:46.913071 21974 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42287/jwks.json
I20260926 14:39:46.918334 21974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:36833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:36833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42287/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:46.919332 21974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:46.920270 21974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:46.928143 21979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:46.928174 21980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:46.928157 21982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:46.928392 21974 server_base.cc:1034] running on GCE node
I20260926 14:39:46.928915 21974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:46.929939 21974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:46.931090 21974 hybrid_clock.cc:648] HybridClock initialized: now 1790433586931075 us; error 29 us; skew 500 ppm
I20260926 14:39:46.932826 21974 webserver.cc:460] Webserver started at http://127.16.90.254:33913/ using document root <none> and password file <none>
I20260926 14:39:46.933362 21974 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:46.933426 21974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:46.933599 21974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:46.935541 21974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "eb44be4dd02845208e5f4e7e73a55023"
format_stamp: "Formatted at 2026-09-26 14:39:46 on dist-test-slave-x4sn"
server_key: "ed4fa574337d9c4a29a32ef1ff911a48"
server_key_iv: "05959b8fab7e3f8d8752623d34f37f18"
server_key_version: "encryptionkey@0"
I20260926 14:39:46.936097 21974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "eb44be4dd02845208e5f4e7e73a55023"
format_stamp: "Formatted at 2026-09-26 14:39:46 on dist-test-slave-x4sn"
server_key: "ed4fa574337d9c4a29a32ef1ff911a48"
server_key_iv: "05959b8fab7e3f8d8752623d34f37f18"
server_key_version: "encryptionkey@0"
I20260926 14:39:46.939616 21974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:39:46.941977 21988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:46.943308 21974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:39:46.943434 21974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "eb44be4dd02845208e5f4e7e73a55023"
format_stamp: "Formatted at 2026-09-26 14:39:46 on dist-test-slave-x4sn"
server_key: "ed4fa574337d9c4a29a32ef1ff911a48"
server_key_iv: "05959b8fab7e3f8d8752623d34f37f18"
server_key_version: "encryptionkey@0"
I20260926 14:39:46.943528 21974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:46.968364 21974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:46.969076 21974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:46.969264 21974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:46.975824 21974 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:36833
I20260926 14:39:46.975834 22040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:36833 every 8 connection(s)
I20260926 14:39:46.976737 21974 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:46.979584 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 21974
I20260926 14:39:46.979737 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:46.979990 22041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:46.980053 16747 external_mini_cluster.cc:1428] Setting key c7658f5e1957b660038904dbd5bb3062
I20260926 14:39:46.986234 22041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023: Bootstrap starting.
I20260926 14:39:46.988487 22041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:46.988642 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:46.981348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.16.90.254:36833)
0926 14:39:46.981995 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:46.982010 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:46.982045 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:39:46.982714 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:46.982731 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:46.983109 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:46.983423 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:46.983874 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:46.984808 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:46.985670 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:46.986122 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:46.986263 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:46.987029 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:46.987034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:46.987045 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:46.987404 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:46.987458 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:46.987477 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:46.987562 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:46.987718 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 14:39:46.987949 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":148,"threads_started":1}
I20260926 14:39:46.989248 22041 log.cc:824] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:46.991189 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:39:46.991319 22041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023: No bootstrap required, opened a new log
I20260926 14:39:46.994726 22041 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb44be4dd02845208e5f4e7e73a55023" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36833 } }
I20260926 14:39:46.994949 22041 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:46.995005 22041 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb44be4dd02845208e5f4e7e73a55023, State: Initialized, Role: FOLLOWER
I20260926 14:39:46.995395 22041 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb44be4dd02845208e5f4e7e73a55023" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36833 } }
I20260926 14:39:46.995524 22041 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:46.995587 22041 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:46.995711 22041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:46.996543 22041 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb44be4dd02845208e5f4e7e73a55023" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36833 } }
I20260926 14:39:46.996865 22041 leader_election.cc:302] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb44be4dd02845208e5f4e7e73a55023; no voters:
I20260926 14:39:46.997176 22041 leader_election.cc:288] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:46.997267 22046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:46.997462 22046 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [term 1 LEADER]: Becoming Leader. State: Replica: eb44be4dd02845208e5f4e7e73a55023, State: Running, Role: LEADER
I20260926 14:39:46.997773 22046 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb44be4dd02845208e5f4e7e73a55023" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36833 } }
I20260926 14:39:46.998322 22041 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:46.999416 22048 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb44be4dd02845208e5f4e7e73a55023. Latest consensus state: current_term: 1 leader_uuid: "eb44be4dd02845208e5f4e7e73a55023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb44be4dd02845208e5f4e7e73a55023" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36833 } } }
I20260926 14:39:46.999532 22048 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:46.999773 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb44be4dd02845208e5f4e7e73a55023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb44be4dd02845208e5f4e7e73a55023" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36833 } } }
I20260926 14:39:46.999861 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:46.999922 22055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:47.002970 22055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:47.008937 22055 catalog_manager.cc:1347] Generated new cluster ID: 0fd1894c8e69418096d2b23e486f3aa5
I20260926 14:39:47.009013 22055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:47.014631 22055 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:47.015846 22055 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:47.020773 22055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb44be4dd02845208e5f4e7e73a55023: Generated new TSK 0
I20260926 14:39:47.021605 22055 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 14:39:47.044965 22071 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 14:39:47.045045 22071 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 14:39:47.045862 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:47.030250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50302 (local address 127.16.90.254:36833)
0926 14:39:47.030375 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:47.030379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:47.030481 (+ 102us) server_negotiation.cc:407] Connection header received
0926 14:39:47.030624 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:47.030627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:47.030683 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:47.030805 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:39:47.030982 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.031802 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.032626 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.032979 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.033083 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:47.033330 (+ 247us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:39:47.045420 (+ 12090us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42287/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 14:39:47.045699 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50302: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42287/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 14:39:47.045921 22070 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:39:47.030153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:36833 (local address 127.0.0.1:50302)
0926 14:39:47.030400 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:47.030412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:47.030504 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:47.030796 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:47.030798 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:47.030803 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:39:47.030896 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:47.030901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.031916 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.031918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:47.032527 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:47.032534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.033075 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.033078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:47.033132 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:47.033153 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:39:47.045611 (+ 12458us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42287/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 14:39:47.045780 (+ 169us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.90.254:36833: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42287/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
W20260926 14:39:47.045938 22044 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50302: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42287/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 14:39:47.046697 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 21974
2026-09-26T14:39:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (290 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 14:39:47.085561 16747 mini_oidc.cc:152] Starting JWKS server
I20260926 14:39:47.086457 16747 webserver.cc:460] Webserver started at https://127.0.0.1:43595/ using document root <none> and password file <none>
I20260926 14:39:47.086567 16747 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 14:39:47.086828 16747 webserver.cc:460] Webserver started at http://127.0.0.1:40449/ using document root <none> and password file <none>
2026-09-26T14:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:47Z Disabled control of system clock
I20260926 14:39:47.095218 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:44929
--webserver_interface=127.16.90.254
--webserver_port=0
--builtin_ntp_servers=127.16.90.212:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:44929
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43595/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 14:39:47.198138 22077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:47.198393 22077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:47.198446 22077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:47.202370 22077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:47.202426 22077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:47.202448 22077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:47.202467 22077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:47.202489 22077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:39:47.202517 22077 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:39:47.202543 22077 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43595/jwks.json
I20260926 14:39:47.207108 22077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:44929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:44929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43595/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:47.208124 22077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:47.209479 22077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:47.217334 22082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:47.217377 22085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:47.217362 22083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:47.217566 22077 server_base.cc:1034] running on GCE node
I20260926 14:39:47.218080 22077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:47.219120 22077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:47.220290 22077 hybrid_clock.cc:648] HybridClock initialized: now 1790433587220255 us; error 47 us; skew 500 ppm
I20260926 14:39:47.222043 22077 webserver.cc:460] Webserver started at http://127.16.90.254:42467/ using document root <none> and password file <none>
I20260926 14:39:47.222586 22077 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:47.222643 22077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:47.222834 22077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:47.224807 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "d9a9faeaa8ce454588fa55885507f4cb"
format_stamp: "Formatted at 2026-09-26 14:39:47 on dist-test-slave-x4sn"
server_key: "dff66f818d0ac976d7245bfb5f58296b"
server_key_iv: "071fe0367fe4ae914bca505abad54884"
server_key_version: "encryptionkey@0"
I20260926 14:39:47.225325 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "d9a9faeaa8ce454588fa55885507f4cb"
format_stamp: "Formatted at 2026-09-26 14:39:47 on dist-test-slave-x4sn"
server_key: "dff66f818d0ac976d7245bfb5f58296b"
server_key_iv: "071fe0367fe4ae914bca505abad54884"
server_key_version: "encryptionkey@0"
I20260926 14:39:47.228799 22077 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:47.231156 22091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:47.232357 22077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:47.232460 22077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "d9a9faeaa8ce454588fa55885507f4cb"
format_stamp: "Formatted at 2026-09-26 14:39:47 on dist-test-slave-x4sn"
server_key: "dff66f818d0ac976d7245bfb5f58296b"
server_key_iv: "071fe0367fe4ae914bca505abad54884"
server_key_version: "encryptionkey@0"
I20260926 14:39:47.232554 22077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:47.245911 22077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:47.246546 22077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:47.246703 22077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:47.253489 22077 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:44929
I20260926 14:39:47.253518 22143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:44929 every 8 connection(s)
I20260926 14:39:47.254367 22077 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:47.257527 22144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:47.259912 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 22077
I20260926 14:39:47.260020 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:47.260267 16747 external_mini_cluster.cc:1428] Setting key f5dc45aba720e35cfd0e71d175720341
I20260926 14:39:47.263778 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb: Bootstrap starting.
I20260926 14:39:47.266397 22144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:47.267220 22144 log.cc:824] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:47.269425 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb: No bootstrap required, opened a new log
I20260926 14:39:47.270808 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:47.261555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.16.90.254:44929)
0926 14:39:47.262240 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:47.262260 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:47.262305 (+ 45us) server_negotiation.cc:407] Connection header received
0926 14:39:47.263107 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:47.263130 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:47.263611 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:47.264024 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:47.264473 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.265942 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.266829 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.267560 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.267750 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:47.268709 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:47.268715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:47.268731 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:47.269232 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:47.269313 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:47.269341 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:47.269442 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:47.269610 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 14:39:47.269885 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":120,"threads_started":1}
I20260926 14:39:47.273689 22144 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a9faeaa8ce454588fa55885507f4cb" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44929 } }
I20260926 14:39:47.273916 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:39:47.273965 22144 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:47.274020 22144 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a9faeaa8ce454588fa55885507f4cb, State: Initialized, Role: FOLLOWER
I20260926 14:39:47.274426 22144 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a9faeaa8ce454588fa55885507f4cb" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44929 } }
I20260926 14:39:47.274557 22144 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:47.274621 22144 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:47.274718 22144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:47.275858 22144 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a9faeaa8ce454588fa55885507f4cb" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44929 } }
I20260926 14:39:47.276178 22144 leader_election.cc:302] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9a9faeaa8ce454588fa55885507f4cb; no voters:
I20260926 14:39:47.276473 22144 leader_election.cc:288] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:47.276598 22149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:47.276840 22149 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [term 1 LEADER]: Becoming Leader. State: Replica: d9a9faeaa8ce454588fa55885507f4cb, State: Running, Role: LEADER
I20260926 14:39:47.277154 22149 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a9faeaa8ce454588fa55885507f4cb" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44929 } }
I20260926 14:39:47.277637 22144 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:47.278548 22150 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9a9faeaa8ce454588fa55885507f4cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a9faeaa8ce454588fa55885507f4cb" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44929 } } }
I20260926 14:39:47.278669 22150 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:47.278956 22150 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a9faeaa8ce454588fa55885507f4cb. Latest consensus state: current_term: 1 leader_uuid: "d9a9faeaa8ce454588fa55885507f4cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a9faeaa8ce454588fa55885507f4cb" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44929 } } }
I20260926 14:39:47.279040 22150 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:47.279311 22158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:47.282174 22158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:47.287110 22158 catalog_manager.cc:1347] Generated new cluster ID: 44ee4c7c57134fa9a2c6e2b0d06b1403
I20260926 14:39:47.287173 22158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:47.341385 22158 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:47.342293 22158 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:47.347380 22158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9a9faeaa8ce454588fa55885507f4cb: Generated new TSK 0
I20260926 14:39:47.348070 22158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:39:47.366252 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:47.361729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.16.90.254:44929)
0926 14:39:47.361859 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:47.361864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:47.362125 (+ 261us) server_negotiation.cc:407] Connection header received
0926 14:39:47.362285 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:47.362287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:47.362338 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:47.362438 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:39:47.362697 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.363217 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.364174 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.364396 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.364503 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:47.364803 (+ 300us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:39:47.365914 (+ 1111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43595/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 14:39:47.366146 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53870: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43595/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 14:39:47.366315 22147 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53870: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43595/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 14:39:47.368947 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:39:47.361645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44929 (local address 127.0.0.1:53870)
0926 14:39:47.362035 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:47.362048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:47.362161 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:47.362451 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:47.362454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:47.362463 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:39:47.362586 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:47.362593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.363340 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.363344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:47.364046 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:47.364055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.364522 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.364527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:47.364605 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:47.364629 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:39:47.366072 (+ 1443us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43595/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 14:39:47.368791 (+ 2719us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.90.254:44929: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43595/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":2540,"thread_start_us":120,"threads_started":1}
I20260926 14:39:47.369572 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 22077
E20260926 14:39:47.379204 22174 webserver.cc:533] Webserver: sslize failed:
E20260926 14:39:47.379272 22174 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T14:39:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (322 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T14:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:47Z Disabled control of system clock
I20260926 14:39:47.415553 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35825
--webserver_interface=127.16.90.254
--webserver_port=0
--builtin_ntp_servers=127.16.90.212:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:35825
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/testchainca.cert with env {}
W20260926 14:39:47.520370 22178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:47.520609 22178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:47.520644 22178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:47.524662 22178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:47.524715 22178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:47.524729 22178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:47.524744 22178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:47.524758 22178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:39:47.524771 22178 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:39:47.524786 22178 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 14:39:47.529995 22178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:35825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:47.530958 22178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:47.531948 22178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:47.539772 22184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:47.539853 22183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:47.539772 22186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:47.539947 22178 server_base.cc:1034] running on GCE node
I20260926 14:39:47.540563 22178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:47.541584 22178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:47.542745 22178 hybrid_clock.cc:648] HybridClock initialized: now 1790433587542731 us; error 26 us; skew 500 ppm
I20260926 14:39:47.544504 22178 webserver.cc:460] Webserver started at http://127.16.90.254:34493/ using document root <none> and password file <none>
I20260926 14:39:47.545019 22178 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:47.545094 22178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:47.545274 22178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:47.547195 22178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "81a9d13288fa4cfebf83fe3e2a57b48a"
format_stamp: "Formatted at 2026-09-26 14:39:47 on dist-test-slave-x4sn"
server_key: "a4c6fd54525946bc0598bf2d6fc8f4fd"
server_key_iv: "9730f44bb7f4294dc3318322fde4de85"
server_key_version: "encryptionkey@0"
I20260926 14:39:47.547737 22178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "81a9d13288fa4cfebf83fe3e2a57b48a"
format_stamp: "Formatted at 2026-09-26 14:39:47 on dist-test-slave-x4sn"
server_key: "a4c6fd54525946bc0598bf2d6fc8f4fd"
server_key_iv: "9730f44bb7f4294dc3318322fde4de85"
server_key_version: "encryptionkey@0"
I20260926 14:39:47.551208 22178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:39:47.553694 22192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:47.554827 22178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:39:47.554937 22178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "81a9d13288fa4cfebf83fe3e2a57b48a"
format_stamp: "Formatted at 2026-09-26 14:39:47 on dist-test-slave-x4sn"
server_key: "a4c6fd54525946bc0598bf2d6fc8f4fd"
server_key_iv: "9730f44bb7f4294dc3318322fde4de85"
server_key_version: "encryptionkey@0"
I20260926 14:39:47.555029 22178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:47.584442 22178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:47.585146 22178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:47.585318 22178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:47.592218 22178 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:35825
I20260926 14:39:47.592245 22245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:35825 every 8 connection(s)
I20260926 14:39:47.593129 22178 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:47.595881 22246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:47.600672 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 22178
I20260926 14:39:47.600791 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:47.601042 16747 external_mini_cluster.cc:1428] Setting key 8eecd77e78736c962fb2950745e2ded7
I20260926 14:39:47.601349 22246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a: Bootstrap starting.
I20260926 14:39:47.603771 22246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:47.604533 22246 log.cc:824] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:47.606524 22246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a: No bootstrap required, opened a new log
I20260926 14:39:47.609853 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:47.602386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.16.90.254:35825)
0926 14:39:47.602977 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:47.602992 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:47.603027 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:39:47.603646 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:47.603662 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:47.604113 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:47.604395 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:47.604748 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.605693 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.606872 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.607325 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.607442 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:47.608263 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:47.608268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:47.608279 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:39:47.608631 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:47.608683 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:39:47.608703 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:47.608774 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:47.608915 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 14:39:47.609164 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260926 14:39:47.610245 22246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a9d13288fa4cfebf83fe3e2a57b48a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35825 } }
I20260926 14:39:47.610456 22246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:47.610518 22246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a9d13288fa4cfebf83fe3e2a57b48a, State: Initialized, Role: FOLLOWER
I20260926 14:39:47.610890 22246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a9d13288fa4cfebf83fe3e2a57b48a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35825 } }
I20260926 14:39:47.611016 22246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:47.611078 22246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:47.611181 22246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:47.612142 22246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a9d13288fa4cfebf83fe3e2a57b48a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35825 } }
I20260926 14:39:47.612469 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:39:47.612478 22246 leader_election.cc:302] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81a9d13288fa4cfebf83fe3e2a57b48a; no voters:
I20260926 14:39:47.612764 22246 leader_election.cc:288] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:47.612859 22251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:47.613042 22251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [term 1 LEADER]: Becoming Leader. State: Replica: 81a9d13288fa4cfebf83fe3e2a57b48a, State: Running, Role: LEADER
I20260926 14:39:47.613329 22251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a9d13288fa4cfebf83fe3e2a57b48a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35825 } }
I20260926 14:39:47.613965 22246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:47.614929 22253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81a9d13288fa4cfebf83fe3e2a57b48a. Latest consensus state: current_term: 1 leader_uuid: "81a9d13288fa4cfebf83fe3e2a57b48a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a9d13288fa4cfebf83fe3e2a57b48a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35825 } } }
I20260926 14:39:47.615072 22253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:47.615434 22260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:47.615983 22252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81a9d13288fa4cfebf83fe3e2a57b48a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a9d13288fa4cfebf83fe3e2a57b48a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35825 } } }
I20260926 14:39:47.616147 22252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:47.618005 22260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:47.623232 22260 catalog_manager.cc:1347] Generated new cluster ID: 695d5e2185184f1bad9ce261563a9698
I20260926 14:39:47.623289 22260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:47.628031 22260 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:47.629105 22260 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:47.642210 22260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81a9d13288fa4cfebf83fe3e2a57b48a: Generated new TSK 0
I20260926 14:39:47.643004 22260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:39:48.088609 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:47.656835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.16.90.254:35825)
0926 14:39:47.656959 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:47.656963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:47.657070 (+ 107us) server_negotiation.cc:407] Connection header received
0926 14:39:47.657202 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:47.657205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:47.657260 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:47.657380 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:39:47.657541 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.658414 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.659290 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.659634 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.659759 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:47.660016 (+ 257us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:39:48.088226 (+428210us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 14:39:48.088487 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53074: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 14:39:48.088683 22249 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53074: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 14:39:48.088845 22275 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:39:47.656735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35825 (local address 127.0.0.1:53074)
0926 14:39:47.656991 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:47.657004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:47.657091 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:47.657367 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:47.657369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:47.657374 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:39:47.657459 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:47.657464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.658514 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.658517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:47.659186 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:47.659195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:47.659769 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:47.659772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:47.659840 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:47.659862 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:39:48.088453 (+428591us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 14:39:48.088634 (+ 181us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.90.254:35825: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":97,"threads_started":1}
I20260926 14:39:48.089371 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 22178
2026-09-26T14:39:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (699 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (117 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:39:48 dist-test-slave-x4sn krb5kdc[22287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:39:48 dist-test-slave-x4sn krb5kdc[22287](info): set up 2 sockets
Sep 26 14:39:48 dist-test-slave-x4sn krb5kdc[22287](info): commencing operation
krb5kdc: starting...
W20260926 14:39:49.380842 16747 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 14:39:49.622603 16747 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.242s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 14:39:49.889339 16747 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.267s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 14:39:50.187780 16747 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.298s user 0.002s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/test-user.keytab.
W20260926 14:39:50.955335 16747 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.767s user 0.001s sys 0.001s
Sep 26 14:39:50 dist-test-slave-x4sn krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:39:50.996692 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:43037
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:43037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:51.102098 22300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:51.102345 22300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:51.102382 22300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:51.106310 22300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:51.106364 22300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:51.106381 22300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:51.106396 22300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:51.106411 22300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:51.111559 22300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:43037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:43037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:51.112524 22300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:51.113533 22300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:51.121536 22306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:51.121629 22300 server_base.cc:1034] running on GCE node
W20260926 14:39:51.121526 22308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:51.121527 22305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:51.122388 22300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:51.123394 22300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:51.124537 22300 hybrid_clock.cc:648] HybridClock initialized: now 1790433591124518 us; error 40 us; skew 500 ppm
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:51.129822 22300 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:39:51.130683 22300 webserver.cc:460] Webserver started at http://127.16.90.254:46367/ using document root <none> and password file <none>
I20260926 14:39:51.131201 22300 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:51.131264 22300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:51.131456 22300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:51.133495 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "4d06576335dc47aaa36e04eb61364f87"
format_stamp: "Formatted at 2026-09-26 14:39:51 on dist-test-slave-x4sn"
server_key: "27391b1bb7bc9025a9e1fb86ca217bc2"
server_key_iv: "0bdc351f4a3bff9ee38c6e389c4d006a"
server_key_version: "encryptionkey@0"
I20260926 14:39:51.134027 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "4d06576335dc47aaa36e04eb61364f87"
format_stamp: "Formatted at 2026-09-26 14:39:51 on dist-test-slave-x4sn"
server_key: "27391b1bb7bc9025a9e1fb86ca217bc2"
server_key_iv: "0bdc351f4a3bff9ee38c6e389c4d006a"
server_key_version: "encryptionkey@0"
I20260926 14:39:51.137603 22300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:39:51.139993 22316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:51.141268 22300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:51.141389 22300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "4d06576335dc47aaa36e04eb61364f87"
format_stamp: "Formatted at 2026-09-26 14:39:51 on dist-test-slave-x4sn"
server_key: "27391b1bb7bc9025a9e1fb86ca217bc2"
server_key_iv: "0bdc351f4a3bff9ee38c6e389c4d006a"
server_key_version: "encryptionkey@0"
I20260926 14:39:51.141496 22300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:51.174805 22300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:51.177932 22300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:51.178150 22300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:51.184965 22300 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:43037
I20260926 14:39:51.184983 22368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:43037 every 8 connection(s)
I20260926 14:39:51.185940 22300 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:51.188630 22369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:51.192492 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 22300
I20260926 14:39:51.192612 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:51.192878 16747 external_mini_cluster.cc:1428] Setting key 0d1331319d96ba0f83cbd1ace00b51e8
I20260926 14:39:51.194360 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87: Bootstrap starting.
I20260926 14:39:51.196666 22369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:51.197423 22369 log.cc:824] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:51.199388 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87: No bootstrap required, opened a new log
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:51.203509 22369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d06576335dc47aaa36e04eb61364f87" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43037 } }
I20260926 14:39:51.203792 22369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:51.203850 22369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d06576335dc47aaa36e04eb61364f87, State: Initialized, Role: FOLLOWER
I20260926 14:39:51.204324 22369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d06576335dc47aaa36e04eb61364f87" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43037 } }
I20260926 14:39:51.204461 22369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:51.204522 22369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:51.204618 22369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:51.205636 22369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d06576335dc47aaa36e04eb61364f87" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43037 } }
I20260926 14:39:51.205981 22369 leader_election.cc:302] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d06576335dc47aaa36e04eb61364f87; no voters:
I20260926 14:39:51.206355 22369 leader_election.cc:288] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:51.206441 22374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:51.206664 22374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [term 1 LEADER]: Becoming Leader. State: Replica: 4d06576335dc47aaa36e04eb61364f87, State: Running, Role: LEADER
I20260926 14:39:51.207016 22374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d06576335dc47aaa36e04eb61364f87" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43037 } }
I20260926 14:39:51.207295 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:51.194284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.16.90.254:43037)
0926 14:39:51.194704 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:51.194720 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:51.194756 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:39:51.195389 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:51.195406 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:51.195784 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:51.196101 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:51.196968 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.197981 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.199114 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.199612 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.199800 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:51.202326 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:51.202332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:51.202348 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:51.202378 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:51.204175 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.204588 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.204593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.204598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.204766 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.205019 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.205022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.205024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.205509 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:51.205705 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:51.205986 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 14:39:51.206303 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":128,"threads_started":1}
I20260926 14:39:51.207553 22369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:51.208544 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d06576335dc47aaa36e04eb61364f87. Latest consensus state: current_term: 1 leader_uuid: "4d06576335dc47aaa36e04eb61364f87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d06576335dc47aaa36e04eb61364f87" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43037 } } }
I20260926 14:39:51.208659 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:51.208962 22375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d06576335dc47aaa36e04eb61364f87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d06576335dc47aaa36e04eb61364f87" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43037 } } }
I20260926 14:39:51.209038 22375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:51.209828 22383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:51.212668 22383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:51.217502 22383 catalog_manager.cc:1347] Generated new cluster ID: efea4cfccd2640209cae7927f8ae72a6
I20260926 14:39:51.217572 22383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:51.226249 22383 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:51.227128 22383 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:51.231621 22383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d06576335dc47aaa36e04eb61364f87: Generated new TSK 0
I20260926 14:39:51.232271 22383 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:39:51.256687 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43037
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:51.362999 22396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:51.363237 22396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:51.363281 22396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:51.367190 22396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:51.367250 22396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:51.367307 22396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:51.372349 22396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:51.373377 22396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:51.374392 22396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:51.383777 22404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:51.383852 22401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:51.383816 22402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:51.384369 22396 server_base.cc:1034] running on GCE node
I20260926 14:39:51.384754 22396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:51.385329 22396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:51.386485 22396 hybrid_clock.cc:648] HybridClock initialized: now 1790433591386462 us; error 34 us; skew 500 ppm
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:51.389088 22396 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:39:51.389899 22396 webserver.cc:460] Webserver started at http://127.16.90.193:36095/ using document root <none> and password file <none>
I20260926 14:39:51.390395 22396 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:51.390448 22396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:51.390612 22396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:51.392633 22396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "f009b2d0338d48be81599122fd4295cf"
format_stamp: "Formatted at 2026-09-26 14:39:51 on dist-test-slave-x4sn"
server_key: "0684ca52824f77c8664212d72af4d65a"
server_key_iv: "b93af05c440e76a5b5213e5d3c78f62f"
server_key_version: "encryptionkey@0"
I20260926 14:39:51.393124 22396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "f009b2d0338d48be81599122fd4295cf"
format_stamp: "Formatted at 2026-09-26 14:39:51 on dist-test-slave-x4sn"
server_key: "0684ca52824f77c8664212d72af4d65a"
server_key_iv: "b93af05c440e76a5b5213e5d3c78f62f"
server_key_version: "encryptionkey@0"
I20260926 14:39:51.396519 22396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:51.398767 22411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:51.399991 22396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:51.400089 22396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "f009b2d0338d48be81599122fd4295cf"
format_stamp: "Formatted at 2026-09-26 14:39:51 on dist-test-slave-x4sn"
server_key: "0684ca52824f77c8664212d72af4d65a"
server_key_iv: "b93af05c440e76a5b5213e5d3c78f62f"
server_key_version: "encryptionkey@0"
I20260926 14:39:51.400177 22396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:51.420130 22396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:51.423014 22396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:51.423170 22396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:51.423746 22396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:51.424535 22396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:51.424582 22396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:51.424620 22396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:51.424640 22396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:51.433071 22396 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:34079
I20260926 14:39:51.433096 22524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:34079 every 8 connection(s)
I20260926 14:39:51.433939 22396 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:39:51.442693 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 22396
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:51.442792 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:39:51.443107 16747 external_mini_cluster.cc:1428] Setting key 2caee078a8655de24c6838fd00defc70
I20260926 14:39:51.446835 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:51.435594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:52301 (local address 127.16.90.254:43037)
0926 14:39:51.435784 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:51.435789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:51.436608 (+ 819us) server_negotiation.cc:407] Connection header received
0926 14:39:51.437883 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:51.437885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:51.437931 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:51.438000 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:51.439093 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.439633 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.440774 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.440997 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.441084 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:51.443404 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:51.443407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:51.443409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:51.443433 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:51.444918 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.445364 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.445368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.445369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.445459 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.445784 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.445786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.445788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.445994 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:51.446086 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:51.446611 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 14:39:51.446718 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 14:39:51.447820 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:51.436014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:43037 (local address 127.16.90.193:52301)
0926 14:39:51.436465 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:51.436508 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:51.437689 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:51.438150 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:51.438157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:51.438494 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:51.438979 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:51.438993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.439783 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.439787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:51.440651 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:51.440660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.441131 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.441134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:51.441297 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:51.441788 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:51.441803 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:51.443242 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:51.445017 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:51.445021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:51.445032 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:51.445270 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:51.445571 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:51.445573 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:51.445575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:51.445709 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:51.446081 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:51.446085 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:51.446377 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 14:39:51.446576 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 14:39:51.446901 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":129,"threads_started":1}
I20260926 14:39:51.449074 22525 heartbeater.cc:347] Connected to a master server at 127.16.90.254:43037
I20260926 14:39:51.449328 22525 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:51.449867 22525 heartbeater.cc:511] Master 127.16.90.254:43037 requested a full tablet report, sending...
I20260926 14:39:51.451572 22333 ts_manager.cc:194] Registered new tserver with Master: f009b2d0338d48be81599122fd4295cf (127.16.90.193:34079)
I20260926 14:39:51.453156 22333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:52301
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:39:51.470655 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43037
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:51.576273 22530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:51.576520 22530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:51.576575 22530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:51.580556 22530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:51.580610 22530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:51.580685 22530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:51.585808 22530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:51.586797 22530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:51.587807 22530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:51.596796 22535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:51.596809 22538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:51.596892 22530 server_base.cc:1034] running on GCE node
W20260926 14:39:51.596800 22536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:51.597457 22530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:51.598058 22530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:51.599218 22530 hybrid_clock.cc:648] HybridClock initialized: now 1790433591599201 us; error 30 us; skew 500 ppm
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:51.601781 22530 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:39:51.602633 22530 webserver.cc:460] Webserver started at http://127.16.90.194:43033/ using document root <none> and password file <none>
I20260926 14:39:51.603195 22530 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:51.603260 22530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:51.603477 22530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:51.605728 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "e5b296d04f6249f7be1642e33f89c225"
format_stamp: "Formatted at 2026-09-26 14:39:51 on dist-test-slave-x4sn"
server_key: "ea53fd9067430da017dac29deb78a79d"
server_key_iv: "4b27efe6a0c8b6d438516fc3e4c9671a"
server_key_version: "encryptionkey@0"
I20260926 14:39:51.606287 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "e5b296d04f6249f7be1642e33f89c225"
format_stamp: "Formatted at 2026-09-26 14:39:51 on dist-test-slave-x4sn"
server_key: "ea53fd9067430da017dac29deb78a79d"
server_key_iv: "4b27efe6a0c8b6d438516fc3e4c9671a"
server_key_version: "encryptionkey@0"
I20260926 14:39:51.610224 22530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:39:51.612663 22546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:51.613848 22530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:51.613955 22530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "e5b296d04f6249f7be1642e33f89c225"
format_stamp: "Formatted at 2026-09-26 14:39:51 on dist-test-slave-x4sn"
server_key: "ea53fd9067430da017dac29deb78a79d"
server_key_iv: "4b27efe6a0c8b6d438516fc3e4c9671a"
server_key_version: "encryptionkey@0"
I20260926 14:39:51.614049 22530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:51.631405 22530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:51.634553 22530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:51.634732 22530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:51.635301 22530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:51.636138 22530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:51.636194 22530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:51.636246 22530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:51.636274 22530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:51.644533 22530 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:33863
I20260926 14:39:51.644556 22659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:33863 every 8 connection(s)
I20260926 14:39:51.645457 22530 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:39:51.646310 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 22530
I20260926 14:39:51.646435 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:39:51.646739 16747 external_mini_cluster.cc:1428] Setting key c079d7ba4d69278a3df0e8b7c1528db7
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
I20260926 14:39:51.660175 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:51.648843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:43477 (local address 127.16.90.254:43037)
0926 14:39:51.648991 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:51.648996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:51.649013 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:39:51.649886 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:51.649889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:51.649941 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:51.650013 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:51.651405 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.652043 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.653742 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.653965 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.654047 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:51.656673 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:51.656676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:51.656678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:51.656705 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:51.658089 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.658572 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.658575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.658577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.658680 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.659044 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.659046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.659047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.659229 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:51.659300 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:51.659896 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 14:39:51.660005 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 14:39:51.661070 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:51.647828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:43037 (local address 127.16.90.194:43477)
0926 14:39:51.648315 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:51.648368 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:51.649685 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:51.650206 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:51.650213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:51.650608 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:51.651251 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:51.651269 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.652165 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.652169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:51.653601 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:51.653614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.654115 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.654120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:51.654323 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:51.654921 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:51.654943 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:51.656511 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:51.658190 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:51.658195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:51.658207 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:51.658473 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:51.658775 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:51.658779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:51.658780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:51.658959 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:51.659321 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:51.659328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:51.659633 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 14:39:51.659855 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 14:39:51.660213 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":122,"threads_started":1}
I20260926 14:39:51.662283 22660 heartbeater.cc:347] Connected to a master server at 127.16.90.254:43037
I20260926 14:39:51.662508 22660 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:51.662992 22660 heartbeater.cc:511] Master 127.16.90.254:43037 requested a full tablet report, sending...
I20260926 14:39:51.664093 22333 ts_manager.cc:194] Registered new tserver with Master: e5b296d04f6249f7be1642e33f89c225 (127.16.90.194:33863)
I20260926 14:39:51.664877 22333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:43477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:39:51.674535 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43037
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:51.782730 22665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:51.782966 22665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:51.783007 22665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:51.787050 22665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:51.787101 22665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:51.787164 22665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:51.792209 22665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:43037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:51.793202 22665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:51.794230 22665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:51.803421 22673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:51.803403 22671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:51.803455 22665 server_base.cc:1034] running on GCE node
W20260926 14:39:51.803407 22670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:51.804145 22665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:51.804766 22665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:51.805928 22665 hybrid_clock.cc:648] HybridClock initialized: now 1790433591805913 us; error 29 us; skew 500 ppm
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:51.808593 22665 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:39:51.809435 22665 webserver.cc:460] Webserver started at http://127.16.90.195:39845/ using document root <none> and password file <none>
I20260926 14:39:51.809923 22665 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:51.809976 22665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:51.810143 22665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:51.812104 22665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "e2e001a755c24db9a0fc8b94cea3a7f3"
format_stamp: "Formatted at 2026-09-26 14:39:51 on dist-test-slave-x4sn"
server_key: "593a991eb21352b7458e93c3d1e6894d"
server_key_iv: "8c3ed64437f1f98c18bf353c0cf8d552"
server_key_version: "encryptionkey@0"
I20260926 14:39:51.812592 22665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "e2e001a755c24db9a0fc8b94cea3a7f3"
format_stamp: "Formatted at 2026-09-26 14:39:51 on dist-test-slave-x4sn"
server_key: "593a991eb21352b7458e93c3d1e6894d"
server_key_iv: "8c3ed64437f1f98c18bf353c0cf8d552"
server_key_version: "encryptionkey@0"
I20260926 14:39:51.815939 22665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:51.818352 22680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:51.819474 22665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:51.819571 22665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "e2e001a755c24db9a0fc8b94cea3a7f3"
format_stamp: "Formatted at 2026-09-26 14:39:51 on dist-test-slave-x4sn"
server_key: "593a991eb21352b7458e93c3d1e6894d"
server_key_iv: "8c3ed64437f1f98c18bf353c0cf8d552"
server_key_version: "encryptionkey@0"
I20260926 14:39:51.819657 22665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:51.841305 22665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:51.843955 22665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:51.844125 22665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:51.844733 22665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:51.845516 22665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:51.845568 22665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:51.845620 22665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:51.845654 22665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:51.853645 22665 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:42651
I20260926 14:39:51.853677 22794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:42651 every 8 connection(s)
I20260926 14:39:51.854462 22665 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:51.860244 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 22665
I20260926 14:39:51.860347 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:39:51.860599 16747 external_mini_cluster.cc:1428] Setting key 7310b3349839789d6fa4b9e9fbcca367
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:39:51.866813 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:51.856071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:45401 (local address 127.16.90.254:43037)
0926 14:39:51.856235 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:51.856239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:51.857018 (+ 779us) server_negotiation.cc:407] Connection header received
0926 14:39:51.858297 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:51.858301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:51.858358 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:51.858430 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:51.859537 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.860162 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.861294 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.861600 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.861691 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:51.863946 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:51.863949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:51.863951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:51.863973 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:51.865246 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.865656 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.865658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.865659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.865732 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.865975 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.865977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.865978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.866157 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:51.866229 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:51.866607 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 14:39:51.866699 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 14:39:51.867548 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:51.856455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:43037 (local address 127.16.90.195:45401)
0926 14:39:51.856873 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:51.856912 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:51.858089 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:51.858597 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:51.858604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:51.858945 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:51.859390 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:51.859403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.860286 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.860289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:51.861200 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:51.861209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.861757 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.861761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:51.861900 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:51.862451 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:51.862469 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:51.863807 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:51.865338 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:51.865342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:51.865352 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:51.865582 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:51.865793 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:51.865795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:51.865796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:51.865910 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:51.866228 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:51.866232 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:51.866436 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 14:39:51.866576 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 14:39:51.866804 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
I20260926 14:39:51.868520 22795 heartbeater.cc:347] Connected to a master server at 127.16.90.254:43037
I20260926 14:39:51.868718 22795 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:51.869117 22795 heartbeater.cc:511] Master 127.16.90.254:43037 requested a full tablet report, sending...
I20260926 14:39:51.870141 22333 ts_manager.cc:194] Registered new tserver with Master: e2e001a755c24db9a0fc8b94cea3a7f3 (127.16.90.195:42651)
I20260926 14:39:51.870729 22333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:45401
I20260926 14:39:51.873950 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/test-admin.keytab.
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:51.888236 16747 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 14:39:51.890870 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:39:51.889496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.16.90.193:34079)
0926 14:39:51.889848 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:51.889858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:51.889878 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:39:51.889923 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:51.889928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:51.890099 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:51.890256 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:51.890684 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40778: BlockingRecv error: recv got EOF from 127.0.0.1:40778 (error 108)
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":141,"threads_started":1}
W20260926 14:39:51.890951 16747 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/krb5cc' of type FILE
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:51.891943 16747 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:51.892771 16747 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:39:51.901288 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:51.893500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.16.90.193:34079)
0926 14:39:51.893626 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:51.893629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:51.893671 (+ 42us) server_negotiation.cc:407] Connection header received
0926 14:39:51.893784 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:51.893786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:51.893823 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:51.893877 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:51.894300 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.895032 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.895891 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.896270 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.896350 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:51.898367 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:51.898376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:51.898388 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:51.898410 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:51.899749 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.900118 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.900122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.900124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.900211 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.900406 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.900409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.900410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.900621 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:51.900786 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:51.900992 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 14:39:51.901131 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 14:39:51.904639 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:39:51.903781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.16.90.193:34079)
0926 14:39:51.903887 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:51.903891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:51.903978 (+ 87us) server_negotiation.cc:407] Connection header received
0926 14:39:51.904083 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:51.904086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:51.904122 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:51.904207 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:51.904541 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40802: BlockingRecv error: recv got EOF from 127.0.0.1:40802 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 14:39:51.904925 16747 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/krb5cc)
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:51.905751 16747 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:51.906476 16747 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:39:51.914499 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:51.907192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.16.90.193:34079)
0926 14:39:51.907305 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:51.907309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:51.907413 (+ 104us) server_negotiation.cc:407] Connection header received
0926 14:39:51.907520 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:51.907522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:51.907555 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:51.907606 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:51.908118 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.908738 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.909611 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.909831 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.909910 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:51.911914 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:51.911918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:51.911920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:51.911947 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:51.913211 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.913553 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.913555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.913556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.913620 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.913826 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.913828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.913829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.913962 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:51.914046 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:51.914270 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 14:39:51.914355 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 14:39:51.916754 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:39:51.916000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.16.90.193:34079)
0926 14:39:51.916081 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:51.916084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:51.916256 (+ 172us) server_negotiation.cc:407] Connection header received
0926 14:39:51.916325 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:51.916326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:51.916360 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:51.916415 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:51.916668 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40810: BlockingRecv error: recv got EOF from 127.0.0.1:40810 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 14:39:51.916864 16747 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433562403030-16747-0/krb5kdc/krb5cc' of type FILE
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:51.917714 16747 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:51.918484 16747 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:39:51.925719 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:51.919189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.16.90.193:34079)
0926 14:39:51.919300 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:51.919303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:51.919334 (+ 31us) server_negotiation.cc:407] Connection header received
0926 14:39:51.919423 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:51.919425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:51.919455 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:51.919513 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:51.919979 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.920448 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.921137 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:51.921343 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:51.921405 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:51.923234 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:51.923237 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:51.923238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:51.923258 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:51.924388 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.924782 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.924784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.924785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.924846 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:51.925084 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:51.925086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:51.925087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:51.925234 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:51.925318 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:51.925512 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 14:39:51.925588 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:39:51.926621 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 22396
I20260926 14:39:51.932209 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 22530
I20260926 14:39:51.936981 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 22665
I20260926 14:39:51.941507 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 22300
2026-09-26T14:39:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3617 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22815](info): set up 2 sockets
Sep 26 14:39:51 dist-test-slave-x4sn krb5kdc[22815](info): commencing operation
krb5kdc: starting...
W20260926 14:39:52.897364 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:39:52 dist-test-slave-x4sn krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.254.keytab.
Entry for principal oryx/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.254.keytab.
I20260926 14:39:53.008834 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:44165
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.254.keytab
--principal=oryx/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:44165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:53.114976 22828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:53.115249 22828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:53.115304 22828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:53.119298 22828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:53.119359 22828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:53.119385 22828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:53.119405 22828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:53.119432 22828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:53.124621 22828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:44165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.254.keytab
--principal=oryx/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:44165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:53.125635 22828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:53.126626 22828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:53.134598 22837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:53.134608 22835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:53.134652 22834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:53.134824 22828 server_base.cc:1034] running on GCE node
I20260926 14:39:53.135385 22828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:53.136512 22828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:53.137665 22828 hybrid_clock.cc:648] HybridClock initialized: now 1790433593137650 us; error 30 us; skew 500 ppm
Sep 26 14:39:53 dist-test-slave-x4sn krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:53.140254 22828 init.cc:377] Logged in from keytab as oryx/127.16.90.254@KRBTEST.COM (short username oryx)
I20260926 14:39:53.141098 22828 webserver.cc:460] Webserver started at http://127.16.90.254:38171/ using document root <none> and password file <none>
I20260926 14:39:53.141620 22828 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:53.141686 22828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:53.141898 22828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:53.143947 22828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "4bc24156c7744341835e117b92b82b36"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "f27b6d42d061b2dd02e3f51a6d984ae6"
server_key_iv: "1b7a8f437bd20aad8df13ac049a4d953"
server_key_version: "encryptionkey@0"
I20260926 14:39:53.144486 22828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "4bc24156c7744341835e117b92b82b36"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "f27b6d42d061b2dd02e3f51a6d984ae6"
server_key_iv: "1b7a8f437bd20aad8df13ac049a4d953"
server_key_version: "encryptionkey@0"
I20260926 14:39:53.148111 22828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:39:53.150357 22844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:53.151503 22828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:53.151612 22828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "4bc24156c7744341835e117b92b82b36"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "f27b6d42d061b2dd02e3f51a6d984ae6"
server_key_iv: "1b7a8f437bd20aad8df13ac049a4d953"
server_key_version: "encryptionkey@0"
I20260926 14:39:53.151726 22828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:53.171746 22828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:53.174546 22828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:53.174722 22828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:53.181438 22828 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:44165
I20260926 14:39:53.181689 22896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:44165 every 8 connection(s)
I20260926 14:39:53.182379 22828 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:53.184357 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 22828
I20260926 14:39:53.184479 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:53.184803 16747 external_mini_cluster.cc:1428] Setting key d8514768fa4b98f728c9df3047b260cc
I20260926 14:39:53.185351 22897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:53.191855 22897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36: Bootstrap starting.
Sep 26 14:39:53 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
I20260926 14:39:53.194564 22897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:53.195395 22897 log.cc:824] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:53.197546 22897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36: No bootstrap required, opened a new log
I20260926 14:39:53.199051 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:53.186090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.16.90.254:44165)
0926 14:39:53.186518 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:53.186537 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:53.186580 (+ 43us) server_negotiation.cc:407] Connection header received
0926 14:39:53.187231 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:53.187247 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:53.187552 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:53.187869 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:53.188701 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.189647 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.190846 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.191512 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.191700 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:53.194195 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:53.194201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:53.194217 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:53.194247 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:53.196067 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:53.196474 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:53.196480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:53.196485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:53.196624 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:53.196886 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:53.196889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:53.196890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:53.197364 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:53.197563 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:53.197836 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 14:39:53.198129 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260926 14:39:53.201351 22897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } }
I20260926 14:39:53.201570 22897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:53.201604 22897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc24156c7744341835e117b92b82b36, State: Initialized, Role: FOLLOWER
I20260926 14:39:53.202001 22897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } }
I20260926 14:39:53.202132 22897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:53.202178 22897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:53.202255 22897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:53.203109 22897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } }
I20260926 14:39:53.203418 22897 leader_election.cc:302] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bc24156c7744341835e117b92b82b36; no voters:
I20260926 14:39:53.203713 22897 leader_election.cc:288] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:53.203850 22902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:53.204058 22902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 1 LEADER]: Becoming Leader. State: Replica: 4bc24156c7744341835e117b92b82b36, State: Running, Role: LEADER
I20260926 14:39:53.204371 22902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } }
I20260926 14:39:53.204833 22897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:53.205895 22904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bc24156c7744341835e117b92b82b36. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } }
I20260926 14:39:53.206023 22904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:53.206387 22911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:53.206378 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } }
I20260926 14:39:53.207753 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:53.209231 22911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:53.215505 22911 catalog_manager.cc:1347] Generated new cluster ID: bd408c3a2c0c4a139220ed59a17cd8ca
I20260926 14:39:53.215576 22911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:53.237560 22911 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:53.238538 22911 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:53.249068 22911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36: Generated new TSK 0
I20260926 14:39:53.249792 22911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab.
Entry for principal oryx/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab.
I20260926 14:39:53.287251 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44165
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab
--principal=oryx/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:53.394294 22923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:53.394547 22923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:53.394606 22923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:53.398540 22923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:53.398605 22923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:53.398686 22923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:53.403929 22923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab
--principal=oryx/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:53.404997 22923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:53.406016 22923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:53.415428 22929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:53.415439 22931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:53.415439 22928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:53.415716 22923 server_base.cc:1034] running on GCE node
I20260926 14:39:53.416194 22923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:53.416827 22923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:53.417975 22923 hybrid_clock.cc:648] HybridClock initialized: now 1790433593417953 us; error 35 us; skew 500 ppm
Sep 26 14:39:53 dist-test-slave-x4sn krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:53.420706 22923 init.cc:377] Logged in from keytab as oryx/127.16.90.193@KRBTEST.COM (short username oryx)
I20260926 14:39:53.421597 22923 webserver.cc:460] Webserver started at http://127.16.90.193:41493/ using document root <none> and password file <none>
I20260926 14:39:53.422134 22923 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:53.422201 22923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:53.422381 22923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:53.424494 22923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "6b35d833c2914abbb9412034f3541486"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "fddd7320e2f5923b6036f3901a69849f"
server_key_iv: "6d1437249b3c33df5c2f28664a218302"
server_key_version: "encryptionkey@0"
I20260926 14:39:53.425027 22923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "6b35d833c2914abbb9412034f3541486"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "fddd7320e2f5923b6036f3901a69849f"
server_key_iv: "6d1437249b3c33df5c2f28664a218302"
server_key_version: "encryptionkey@0"
I20260926 14:39:53.428589 22923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:53.430859 22938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:53.432087 22923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:53.432200 22923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "6b35d833c2914abbb9412034f3541486"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "fddd7320e2f5923b6036f3901a69849f"
server_key_iv: "6d1437249b3c33df5c2f28664a218302"
server_key_version: "encryptionkey@0"
I20260926 14:39:53.432291 22923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:53.448190 22923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:53.451213 22923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:53.451385 22923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:53.451965 22923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:53.452745 22923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:53.452793 22923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:53.452843 22923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:53.452870 22923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:53.461068 22923 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:39189
I20260926 14:39:53.461095 23052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:39189 every 8 connection(s)
I20260926 14:39:53.461951 22923 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:39:53.463253 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 22923
I20260926 14:39:53.463354 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:39:53.463728 16747 external_mini_cluster.cc:1428] Setting key d7f7590ac8dfb8114a1cd9ba3043aeb5
Sep 26 14:39:53 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.90.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.194@KRBTEST.COM" created.
I20260926 14:39:53.476094 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:53.463742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:40017 (local address 127.16.90.254:44165)
0926 14:39:53.464145 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:53.464149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:53.464788 (+ 639us) server_negotiation.cc:407] Connection header received
0926 14:39:53.466130 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:53.466133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:53.466184 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:53.466272 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:53.467792 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.468555 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.469784 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.470140 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.470259 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:53.472896 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:53.472901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:53.472904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:53.472935 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:53.474173 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:53.474612 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:53.474615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:53.474616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:53.474704 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:53.474979 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:53.474981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:53.474982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:53.475145 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:53.475220 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:53.475811 (+ 591us) server_negotiation.cc:299] Negotiation successful
0926 14:39:53.475939 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316}
I20260926 14:39:53.476917 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:53.464137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.193:40017)
0926 14:39:53.464627 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:53.464689 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:53.465923 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:53.466477 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:53.466486 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:53.466968 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:53.467625 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:53.467642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.468738 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.468741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:53.469651 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:53.469662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.470288 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.470290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:53.470440 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:53.471094 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:53.471120 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:53.472720 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:53.474279 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:53.474284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:53.474295 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:53.474530 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:53.474776 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:53.474778 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:53.474780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:53.474908 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:53.475254 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:53.475261 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:53.475529 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 14:39:53.475741 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 14:39:53.476005 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260926 14:39:53.478099 23053 heartbeater.cc:347] Connected to a master server at 127.16.90.254:44165
I20260926 14:39:53.478319 23053 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:53.478782 23053 heartbeater.cc:511] Master 127.16.90.254:44165 requested a full tablet report, sending...
I20260926 14:39:53.480440 22861 ts_manager.cc:194] Registered new tserver with Master: 6b35d833c2914abbb9412034f3541486 (127.16.90.193:39189)
I20260926 14:39:53.482031 22861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:40017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab.
Entry for principal oryx/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab.
I20260926 14:39:53.491423 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44165
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab
--principal=oryx/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:53.598953 23058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:53.599210 23058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:53.599254 23058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:53.603148 23058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:53.603212 23058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:53.603269 23058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:53.608615 23058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab
--principal=oryx/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:53.609560 23058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:53.610491 23058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:53.619522 23064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:53.619568 23063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:53.619522 23066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:53.619856 23058 server_base.cc:1034] running on GCE node
I20260926 14:39:53.620239 23058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:53.620836 23058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:53.621999 23058 hybrid_clock.cc:648] HybridClock initialized: now 1790433593621976 us; error 33 us; skew 500 ppm
Sep 26 14:39:53 dist-test-slave-x4sn krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:53.624521 23058 init.cc:377] Logged in from keytab as oryx/127.16.90.194@KRBTEST.COM (short username oryx)
I20260926 14:39:53.625324 23058 webserver.cc:460] Webserver started at http://127.16.90.194:38091/ using document root <none> and password file <none>
I20260926 14:39:53.625830 23058 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:53.625887 23058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:53.626073 23058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:53.628073 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "f22762d682134b62bbbcb5e765dbb000"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "4add087f48a1038d31ec8f3fcb4391f4"
server_key_iv: "c90274a941511f78527428c7b91e41d7"
server_key_version: "encryptionkey@0"
I20260926 14:39:53.628587 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "f22762d682134b62bbbcb5e765dbb000"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "4add087f48a1038d31ec8f3fcb4391f4"
server_key_iv: "c90274a941511f78527428c7b91e41d7"
server_key_version: "encryptionkey@0"
I20260926 14:39:53.631992 23058 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:39:53.634294 23073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:53.635476 23058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:53.635578 23058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "f22762d682134b62bbbcb5e765dbb000"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "4add087f48a1038d31ec8f3fcb4391f4"
server_key_iv: "c90274a941511f78527428c7b91e41d7"
server_key_version: "encryptionkey@0"
I20260926 14:39:53.635700 23058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:53.660880 23058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:53.663610 23058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:53.663796 23058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:53.664358 23058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:53.665136 23058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:53.665180 23058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:53.665220 23058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:53.665248 23058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:53.673610 23058 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:34647
I20260926 14:39:53.673632 23187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:34647 every 8 connection(s)
I20260926 14:39:53.674448 23058 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:39:53.677448 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 23058
I20260926 14:39:53.677548 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:39:53.677829 16747 external_mini_cluster.cc:1428] Setting key 60f72255628b29a71bc6a515e169bbde
Sep 26 14:39:53 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.90.195@KRBTEST.COM; defaulting to no policy
I20260926 14:39:53.687896 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:53.676155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:36151 (local address 127.16.90.254:44165)
0926 14:39:53.676310 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:53.676314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:53.677147 (+ 833us) server_negotiation.cc:407] Connection header received
0926 14:39:53.678532 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:53.678535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:53.678587 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:53.678659 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:53.679930 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.680445 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.681588 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.681919 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.682019 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:53.684449 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:53.684453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:53.684456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:53.684483 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:53.685956 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:53.686432 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:53.686436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:53.686438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:53.686526 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:53.686842 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:53.686845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:53.686846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:53.687028 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:53.687116 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:53.687627 (+ 511us) server_negotiation.cc:299] Negotiation successful
0926 14:39:53.687777 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 14:39:53.688673 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:53.676551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.194:36151)
0926 14:39:53.676998 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:53.677046 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:53.678336 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:53.678848 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:53.678855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:53.679289 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:53.679796 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:53.679811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.680587 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.680591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:53.681476 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:53.681486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.682096 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.682099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:53.682253 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:53.682820 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:53.682836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:53.684277 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:53.686079 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:53.686084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:53.686096 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:53.686338 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:53.686609 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:53.686611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:53.686612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:53.686736 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:53.687112 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:53.687117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:53.687343 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 14:39:53.687526 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 14:39:53.687845 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.195@KRBTEST.COM" created.
I20260926 14:39:53.689854 23188 heartbeater.cc:347] Connected to a master server at 127.16.90.254:44165
I20260926 14:39:53.690088 23188 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:53.690552 23188 heartbeater.cc:511] Master 127.16.90.254:44165 requested a full tablet report, sending...
I20260926 14:39:53.691509 22861 ts_manager.cc:194] Registered new tserver with Master: f22762d682134b62bbbcb5e765dbb000 (127.16.90.194:34647)
I20260926 14:39:53.692076 22861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:36151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab.
Entry for principal oryx/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab.
I20260926 14:39:53.701875 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44165
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab
--principal=oryx/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:53.808300 23193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:53.808537 23193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:53.808590 23193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:53.812510 23193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:53.812562 23193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:53.812638 23193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:53.817655 23193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab
--principal=oryx/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:53.818593 23193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:53.819518 23193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:53.828727 23201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:53.828781 23193 server_base.cc:1034] running on GCE node
W20260926 14:39:53.828852 23198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:53.829061 23199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:53.829440 23193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:53.830041 23193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:53.831202 23193 hybrid_clock.cc:648] HybridClock initialized: now 1790433593831180 us; error 33 us; skew 500 ppm
Sep 26 14:39:53 dist-test-slave-x4sn krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:53.836984 23193 init.cc:377] Logged in from keytab as oryx/127.16.90.195@KRBTEST.COM (short username oryx)
I20260926 14:39:53.837805 23193 webserver.cc:460] Webserver started at http://127.16.90.195:39317/ using document root <none> and password file <none>
I20260926 14:39:53.838299 23193 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:53.838352 23193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:53.838569 23193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:53.840633 23193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "603b1dae66d04c5baa64e503a0ae74d1"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "414cb20a53d9acb2241d8aa2a8d13639"
server_key_iv: "31713f4e66f43db092411412d1822c07"
server_key_version: "encryptionkey@0"
I20260926 14:39:53.841111 23193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "603b1dae66d04c5baa64e503a0ae74d1"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "414cb20a53d9acb2241d8aa2a8d13639"
server_key_iv: "31713f4e66f43db092411412d1822c07"
server_key_version: "encryptionkey@0"
I20260926 14:39:53.844506 23193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:39:53.846694 23208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:53.847832 23193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:53.847932 23193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "603b1dae66d04c5baa64e503a0ae74d1"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "414cb20a53d9acb2241d8aa2a8d13639"
server_key_iv: "31713f4e66f43db092411412d1822c07"
server_key_version: "encryptionkey@0"
I20260926 14:39:53.848023 23193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:53.866035 23193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:53.869035 23193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:53.869206 23193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:53.869750 23193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:53.870509 23193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:53.870553 23193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:53.870594 23193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:53.870623 23193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:53.878619 23193 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:44307
I20260926 14:39:53.878661 23321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:44307 every 8 connection(s)
I20260926 14:39:53.879446 23193 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:53.887320 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 23193
I20260926 14:39:53.887442 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:39:53.887776 16747 external_mini_cluster.cc:1428] Setting key 6b66982079f386980e37a08882fb1c13
Sep 26 14:39:53 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
I20260926 14:39:53.897154 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:53.882843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:35155 (local address 127.16.90.254:44165)
0926 14:39:53.882959 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:53.882963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:53.883905 (+ 942us) server_negotiation.cc:407] Connection header received
0926 14:39:53.885713 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:53.885717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:53.885778 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:53.885870 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:53.887558 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.888432 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.890044 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.890392 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.890497 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:53.893125 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:53.893129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:53.893133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:53.893163 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:53.894891 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:53.895452 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:53.895456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:53.895458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:53.895552 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:53.895912 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:53.895916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:53.895918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:53.896151 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:53.896242 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:53.896858 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 14:39:53.896991 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:39:53.898129 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:53.881751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.195:35155)
0926 14:39:53.882439 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:53.883765 (+ 1326us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:53.885443 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:53.886078 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:53.886089 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:53.886620 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:53.887354 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:53.887374 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.888568 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.888572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:53.889933 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:53.889944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:53.890529 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:53.890532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:53.890679 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:53.891235 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:53.891251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:53.892946 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:53.894993 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:53.894997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:53.895010 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:53.895345 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:53.895630 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:53.895633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:53.895635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:53.895826 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:53.896251 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:53.896257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:53.896570 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 14:39:53.896770 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 14:39:53.897112 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":126,"threads_started":1}
I20260926 14:39:53.899319 23322 heartbeater.cc:347] Connected to a master server at 127.16.90.254:44165
I20260926 14:39:53.899586 23322 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:53.900188 23322 heartbeater.cc:511] Master 127.16.90.254:44165 requested a full tablet report, sending...
I20260926 14:39:53.901330 22861 ts_manager.cc:194] Registered new tserver with Master: 603b1dae66d04c5baa64e503a0ae74d1 (127.16.90.195:44307)
I20260926 14:39:53.902061 22861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:35155
I20260926 14:39:53.903818 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.90.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.253.keytab.
Entry for principal oryx/127.16.90.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.253.keytab.
I20260926 14:39:53.932719 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.253:38709
--webserver_interface=127.16.90.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.253.keytab
--principal=oryx/127.16.90.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:44165,127.16.90.253:38709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:54.043041 23327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:54.043321 23327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:54.043377 23327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:54.047354 23327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:54.047420 23327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:54.047446 23327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:54.047465 23327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:54.047492 23327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:54.053162 23327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:44165,127.16.90.253:38709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.253.keytab
--principal=oryx/127.16.90.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.253:38709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.90.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
W20260926 14:39:54.053408 23327 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.90.254:44165,127.16.90.253:38709'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 14:39:54.054288 23327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:54.055321 23327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:54.063576 23333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:54.063576 23332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:54.063611 23335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:54.063841 23327 server_base.cc:1034] running on GCE node
I20260926 14:39:54.064363 23327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:54.065372 23327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:54.066515 23327 hybrid_clock.cc:648] HybridClock initialized: now 1790433594066485 us; error 44 us; skew 500 ppm
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:54.069132 23327 init.cc:377] Logged in from keytab as oryx/127.16.90.253@KRBTEST.COM (short username oryx)
I20260926 14:39:54.069998 23327 webserver.cc:460] Webserver started at http://127.16.90.253:34965/ using document root <none> and password file <none>
I20260926 14:39:54.070508 23327 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:54.070570 23327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:54.070758 23327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:54.072733 23327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data/instance:
uuid: "16b721197e224140a9783e64705f54b6"
format_stamp: "Formatted at 2026-09-26 14:39:54 on dist-test-slave-x4sn"
server_key: "5acf971818166e5c6240cd67691835a6"
server_key_iv: "370308bd581b59b921282eae10153881"
server_key_version: "encryptionkey@0"
I20260926 14:39:54.073225 23327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/wal/instance:
uuid: "16b721197e224140a9783e64705f54b6"
format_stamp: "Formatted at 2026-09-26 14:39:54 on dist-test-slave-x4sn"
server_key: "5acf971818166e5c6240cd67691835a6"
server_key_iv: "370308bd581b59b921282eae10153881"
server_key_version: "encryptionkey@0"
I20260926 14:39:54.076725 23327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:54.078819 23342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:54.079977 23327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:54.080084 23327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/wal
uuid: "16b721197e224140a9783e64705f54b6"
format_stamp: "Formatted at 2026-09-26 14:39:54 on dist-test-slave-x4sn"
server_key: "5acf971818166e5c6240cd67691835a6"
server_key_iv: "370308bd581b59b921282eae10153881"
server_key_version: "encryptionkey@0"
I20260926 14:39:54.080168 23327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:54.103123 23327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:54.106281 23327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:54.106453 23327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:54.113344 23327 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.253:38709
I20260926 14:39:54.113583 23394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.253:38709 every 8 connection(s)
I20260926 14:39:54.114283 23327 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
I20260926 14:39:54.117142 23395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:54.118680 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 23327
I20260926 14:39:54.121655 23395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.90.253@KRBTEST.COM
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.253@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
I20260926 14:39:54.133872 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.124086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.16.90.254:44165)
0926 14:39:54.124234 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.124239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.124560 (+ 321us) server_negotiation.cc:407] Connection header received
0926 14:39:54.124715 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.124717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.124765 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.124841 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.125610 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.126479 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.127751 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.128030 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.128148 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.130597 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.130600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.130602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.130632 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.131963 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.132484 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.132487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.132489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.132574 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.132823 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.132826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.132828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.133096 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.133193 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.133582 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.133711 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 14:39:54.134654 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.124040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:51684)
0926 14:39:54.124398 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.124446 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.124610 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.124894 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.124899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.125302 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.125484 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.125491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.126611 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.126614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.127614 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.127622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.128184 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.128189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.128282 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.128740 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.128756 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.130453 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.132066 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.132074 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.132080 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.132345 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.132671 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.132673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.132674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.132747 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.133228 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.133233 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.133351 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.133528 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.133846 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":33,"thread_start_us":129,"threads_started":1}
I20260926 14:39:54.134660 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.120368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.16.90.253:38709)
0926 14:39:54.121009 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.121035 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.121147 (+ 112us) server_negotiation.cc:407] Connection header received
0926 14:39:54.122108 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.122127 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.122566 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.122924 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.123907 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.125382 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.126588 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.127259 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.127404 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.130014 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.130021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.130032 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.130058 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.131616 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.132042 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.132046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.132051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.132162 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.132415 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.132417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.132419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.132873 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.133592 (+ 719us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.133689 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.133918 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":337,"thread_start_us":145,"threads_started":1}
I20260926 14:39:54.136742 23395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.253@KRBTEST.COM for oryx/127.16.90.253@KRBTEST.COM
I20260926 14:39:54.143961 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.137146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:38709 (local address 127.0.0.1:52834)
0926 14:39:54.137239 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.137252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.137319 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.137484 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.137486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.137602 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.137698 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.137703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.138642 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.138644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.139186 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.139194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.139600 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.139602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.139710 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.140110 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.140120 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.141224 (+ 1104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.142692 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.142694 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.142695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.142860 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.143099 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.143101 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.143102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.143169 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.143480 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.143483 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.143572 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.143663 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.143800 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 14:39:54.143970 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.137224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52834 (local address 127.16.90.253:38709)
0926 14:39:54.137293 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.137296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.137306 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:39:54.137380 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.137382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.137413 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.137467 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.137771 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.138563 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.139297 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.139516 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.139601 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.141339 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.141342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.141344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.141363 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.142601 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.142930 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.142932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.142933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.142999 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.143230 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.143231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.143233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.143381 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.143450 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.143766 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.143853 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 14:39:54.146667 23395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: Bootstrap starting.
I20260926 14:39:54.148892 23395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:54.149595 23395 log.cc:824] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:54.151412 23395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: No bootstrap required, opened a new log
I20260926 14:39:54.154294 23395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } }
I20260926 14:39:54.154523 23395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:54.154594 23395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b721197e224140a9783e64705f54b6, State: Initialized, Role: FOLLOWER
I20260926 14:39:54.155014 23395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } }
I20260926 14:39:54.155485 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } } }
I20260926 14:39:54.155604 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:39:54.156239 23395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 14:39:54.160451 23412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:39:54.160518 23412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:39:54.162794 23327 master_runner.cc:428] Detected that this master 16b721197e224140a9783e64705f54b6 is joining an existing cluster
I20260926 14:39:54.162842 23327 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.90.253:38709
I20260926 14:39:54.171435 23401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:54.171609 23401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } }
I20260926 14:39:54.171962 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.164755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.16.90.254:44165)
0926 14:39:54.164858 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.164862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.165064 (+ 202us) server_negotiation.cc:407] Connection header received
0926 14:39:54.165227 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.165230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.165275 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.165338 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.165775 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.166291 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.167075 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.167289 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.167372 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.168541 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.168545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.168548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.168571 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.170240 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.170675 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.170687 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.170688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.170764 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.171028 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.171031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.171033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.171259 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.171321 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.171693 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.171803 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 14:39:54.172466 23418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.164718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:51692)
0926 14:39:54.164989 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.165002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.165136 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.165366 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.165369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.165572 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.165671 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.165676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.166385 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.166388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.166972 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.166980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.167392 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.167396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.167478 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.167956 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.167973 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.168422 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.170360 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.170363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.170366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.170593 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.170842 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.170845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.170847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.170929 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.171389 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.171392 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.171489 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.172141 (+ 652us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.172302 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"spinlock_wait_cycles":2304,"thread_start_us":102,"threads_started":1}
I20260926 14:39:54.172483 23401 leader_election.cc:288] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc24156c7744341835e117b92b82b36 (127.16.90.254:44165)
I20260926 14:39:54.173472 22871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16b721197e224140a9783e64705f54b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc24156c7744341835e117b92b82b36" is_pre_election: true
I20260926 14:39:54.174212 23345 leader_election.cc:302] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 16b721197e224140a9783e64705f54b6; no voters: 4bc24156c7744341835e117b92b82b36
I20260926 14:39:54.175113 23401 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 14:39:54.175328 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.165023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.16.90.253:38709)
0926 14:39:54.165122 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.165126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.167976 (+ 2850us) server_negotiation.cc:407] Connection header received
0926 14:39:54.168081 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.168084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.168124 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.168196 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.168704 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.169344 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.170511 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.170853 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.170948 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.172424 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.172428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.172430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.172449 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.173929 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.174334 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.174337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.174339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.174408 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.174634 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.174636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.174638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.174785 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.174864 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.175102 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.175201 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:39:54.175885 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.164753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:38709 (local address 127.0.0.1:52846)
0926 14:39:54.167882 (+ 3129us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.167897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.167994 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.168227 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.168230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.168464 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.168587 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.168594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.169458 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.169461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.170404 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.170412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.170966 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.170969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.171067 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.171485 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.171498 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.172310 (+ 812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.174024 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.174027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.174028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.174239 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.174493 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.174495 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.174497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.174567 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.174868 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.174870 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.174944 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.175573 (+ 629us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.175733 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3053,"spinlock_wait_cycles":1408,"thread_start_us":99,"threads_started":1}
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.254@KRBTEST.COM for oryx/127.16.90.253@KRBTEST.COM
I20260926 14:39:54.187630 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.177480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:38709 (local address 127.0.0.1:52858)
0926 14:39:54.177816 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.177864 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.177978 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.178504 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.178522 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.178848 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.178965 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.178971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.179992 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.179995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.180649 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.180658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.181234 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.181239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.181360 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.181987 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.182004 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.183809 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.185671 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.185681 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.185687 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.185977 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.186260 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.186263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.186265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.186356 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.186974 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.186984 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.187116 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.187263 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.187454 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":84,"threads_started":1}
I20260926 14:39:54.188143 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.178132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.16.90.253:38709)
0926 14:39:54.178247 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.178251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.178265 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:39:54.178324 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.178327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.178372 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.178445 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.179063 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.179836 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.180761 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.181087 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.181183 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.183931 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.183936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.183939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.183971 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.185535 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.186076 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.186080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.186082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.186172 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.186440 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.186443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.186445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.186673 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.186786 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.187893 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.188002 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:39:54.189421 22861 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.90.253:38709: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "16b721197e224140a9783e64705f54b6"
member_type: NON_VOTER
last_known_addr {
host: "127.16.90.253"
port: 38709
}
attrs {
promote: true
}
}
dest_uuid: "4bc24156c7744341835e117b92b82b36"
cas_config_opid_index: -1
I20260926 14:39:54.190094 22861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } }
I20260926 14:39:54.192787 23369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:54.194202 23369 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 FOLLOWER]: Refusing update from remote peer 4bc24156c7744341835e117b92b82b36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 14:39:54.195016 23424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:54.195034 23423 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 16b721197e224140a9783e64705f54b6 (127.16.90.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } }
I20260926 14:39:54.201884 23369 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 16b721197e224140a9783e64705f54b6 (127.16.90.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } }
I20260926 14:39:54.202477 23425 mvcc.cc:205] Tried to move back new op lower bound from 7333615997944111104 to 7333615997765885952. Current Snapshot: MvccSnapshot[applied={T|T < 7333615997799129088}]
WARNING: no policy specified for oryx/127.16.90.252@KRBTEST.COM; defaulting to no policy
I20260926 14:39:54.203409 23423 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.90.253:38709
I20260926 14:39:54.203927 23327 master.cc:537] Master@127.16.90.253:38709 shutting down...
I20260926 14:39:54.204581 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } }
I20260926 14:39:54.204718 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.252@KRBTEST.COM" created.
I20260926 14:39:54.206548 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bc24156c7744341835e117b92b82b36. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } }
I20260926 14:39:54.206664 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:39:54.206743 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } }
I20260926 14:39:54.206789 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:39:54.208074 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } }
I20260926 14:39:54.208264 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:39:54.209177 23327 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 LEARNER]: Raft consensus shutting down.
I20260926 14:39:54.209478 23327 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 LEARNER]: Raft consensus is shut down!
I20260926 14:39:54.209547 23327 tablet_replica.cc:330] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.252.keytab.
Entry for principal oryx/127.16.90.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.252.keytab.
I20260926 14:39:54.222425 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.252:45751
--webserver_interface=127.16.90.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.252.keytab
--principal=oryx/127.16.90.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:44165,127.16.90.253:38709,127.16.90.252:45751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:39:54.225896 23327 master.cc:552] Master@127.16.90.253:38709 shutdown complete.
I20260926 14:39:54.225962 23327 master_runner.cc:305] Clearing existing system tablet
I20260926 14:39:54.226221 23327 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:54.226276 23327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:54.227574 23327 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:39:54.228511 23436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:54.228760 23327 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:54.228888 23327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:54.228920 23327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/wal
uuid: "16b721197e224140a9783e64705f54b6"
format_stamp: "Formatted at 2026-09-26 14:39:54 on dist-test-slave-x4sn"
server_key: "5acf971818166e5c6240cd67691835a6"
server_key_iv: "370308bd581b59b921282eae10153881"
server_key_version: "encryptionkey@0"
I20260926 14:39:54.230759 23327 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:54.233222 23327 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 14:39:54.233310 23327 log.cc:1196] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 14:39:54.233589 23327 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: Deleting consensus metadata
I20260926 14:39:54.233741 23327 master_runner.cc:315] Copying system tablet from 127.16.90.254:44165
I20260926 14:39:54.234884 23327 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: tablet copy: Beginning tablet copy session from remote peer at address 127.16.90.254:44165
I20260926 14:39:54.243908 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.235505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:51696)
0926 14:39:54.235915 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.235930 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.236052 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.236343 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.236346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.236552 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.236688 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.236695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.237753 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.237757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.238747 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.238759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.239358 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.239361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.239458 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.240076 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.240090 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.240508 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.242366 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.242369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.242372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.242614 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.242883 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.242886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.242888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.242972 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.243350 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.243353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.243450 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.243553 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.243730 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":81,"threads_started":1}
I20260926 14:39:54.243914 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.235616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.16.90.254:44165)
0926 14:39:54.235762 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.235767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.236010 (+ 243us) server_negotiation.cc:407] Connection header received
0926 14:39:54.236144 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.236147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.236223 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.236319 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.236809 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.237634 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.238870 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.239251 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.239349 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.240667 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.240671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.240674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.240700 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.242257 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.242713 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.242716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.242718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.242803 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.243048 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.243050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.243052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.243256 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.243349 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.243618 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.243758 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 14:39:54.244952 22881 tablet_copy_service.cc:140] P 4bc24156c7744341835e117b92b82b36: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 16b721197e224140a9783e64705f54b6 ({username='oryx', principal='oryx/127.16.90.253@KRBTEST.COM'} at 127.0.0.1:51696)
I20260926 14:39:54.245096 22881 tablet_copy_service.cc:161] P 4bc24156c7744341835e117b92b82b36: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 16b721197e224140a9783e64705f54b6 at {username='oryx', principal='oryx/127.16.90.253@KRBTEST.COM'} at 127.0.0.1:51696: session id = 16b721197e224140a9783e64705f54b6-00000000000000000000000000000000
I20260926 14:39:54.246801 22881 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 14:39:54.248409 23327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 14:39:54.250617 23327 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: tablet copy: Starting download of 0 data blocks...
I20260926 14:39:54.250790 23327 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: tablet copy: Starting download of 1 WAL segments...
I20260926 14:39:54.252561 23327 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 14:39:54.254154 22881 tablet_copy_service.cc:342] P 4bc24156c7744341835e117b92b82b36: Request end of tablet copy session 16b721197e224140a9783e64705f54b6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.90.253@KRBTEST.COM'} at 127.0.0.1:51696
I20260926 14:39:54.254259 22881 tablet_copy_service.cc:433] P 4bc24156c7744341835e117b92b82b36: ending tablet copy session 16b721197e224140a9783e64705f54b6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 16b721197e224140a9783e64705f54b6
I20260926 14:39:54.261461 23327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:54.263022 23446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:54.263360 23443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:54.263480 23327 server_base.cc:1034] running on GCE node
W20260926 14:39:54.263818 23444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:54.264048 23327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:54.264324 23327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:54.265467 23327 hybrid_clock.cc:648] HybridClock initialized: now 1790433594265454 us; error 30 us; skew 500 ppm
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:54.267041 23327 init.cc:377] Logged in from keytab as oryx/127.16.90.253@KRBTEST.COM (short username oryx)
I20260926 14:39:54.267499 23327 webserver.cc:460] Webserver started at http://127.16.90.253:41411/ using document root <none> and password file <none>
I20260926 14:39:54.267720 23327 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:54.267762 23327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:54.268989 23327 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:54.269912 23453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:54.270175 23327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:54.270246 23327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/wal
uuid: "16b721197e224140a9783e64705f54b6"
format_stamp: "Formatted at 2026-09-26 14:39:54 on dist-test-slave-x4sn"
server_key: "5acf971818166e5c6240cd67691835a6"
server_key_iv: "370308bd581b59b921282eae10153881"
server_key_version: "encryptionkey@0"
I20260926 14:39:54.270320 23327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:54.304313 23327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:54.306418 23327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:54.311105 23327 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.253:38709
I20260926 14:39:54.311743 23327 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
I20260926 14:39:54.311834 23505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.253:38709 every 8 connection(s)
I20260926 14:39:54.313491 23506 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:39:54.314540 23506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: Bootstrap starting.
I20260926 14:39:54.321823 23506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:39:54.322093 23506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: Bootstrap complete.
I20260926 14:39:54.322650 23506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } }
I20260926 14:39:54.322772 23506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16b721197e224140a9783e64705f54b6, State: Initialized, Role: LEARNER
I20260926 14:39:54.322861 23506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } }
I20260926 14:39:54.323171 23509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } }
I20260926 14:39:54.323262 23506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:54.323385 23509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:39:54.326257 23520 catalog_manager.cc:1259] Loaded cluster ID: bd408c3a2c0c4a139220ed59a17cd8ca
I20260926 14:39:54.326313 23520 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: loading cluster ID for follower catalog manager: success
I20260926 14:39:54.328516 23520 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: acquiring CA information for follower catalog manager: success
I20260926 14:39:54.329819 23520 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 14:39:54.355369 23434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:54.355625 23434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:54.355683 23434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:54.359911 23434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:54.359969 23434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:54.359987 23434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:54.360002 23434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:54.360015 23434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:54.365242 23434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:44165,127.16.90.253:38709,127.16.90.252:45751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.252.keytab
--principal=oryx/127.16.90.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.252:45751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.90.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:54.366264 23434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:54.367235 23434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:54.375295 23525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:54.375330 23526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:54.375331 23528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:54.375548 23434 server_base.cc:1034] running on GCE node
I20260926 14:39:54.376108 23434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:54.377179 23434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:54.378341 23434 hybrid_clock.cc:648] HybridClock initialized: now 1790433594378322 us; error 32 us; skew 500 ppm
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:54.380949 23434 init.cc:377] Logged in from keytab as oryx/127.16.90.252@KRBTEST.COM (short username oryx)
I20260926 14:39:54.381841 23434 webserver.cc:460] Webserver started at http://127.16.90.252:42971/ using document root <none> and password file <none>
I20260926 14:39:54.382375 23434 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:54.382444 23434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:54.382632 23434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:54.384711 23434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data/instance:
uuid: "2585d75533f343b7a8e38be8e88d7cc1"
format_stamp: "Formatted at 2026-09-26 14:39:54 on dist-test-slave-x4sn"
server_key: "c84861d98a59d901d642f3aae488beaf"
server_key_iv: "9403a3ed8de0796aa4242cb2e0501919"
server_key_version: "encryptionkey@0"
I20260926 14:39:54.385231 23434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/wal/instance:
uuid: "2585d75533f343b7a8e38be8e88d7cc1"
format_stamp: "Formatted at 2026-09-26 14:39:54 on dist-test-slave-x4sn"
server_key: "c84861d98a59d901d642f3aae488beaf"
server_key_iv: "9403a3ed8de0796aa4242cb2e0501919"
server_key_version: "encryptionkey@0"
I20260926 14:39:54.388916 23434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:39:54.391259 23535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:54.392474 23434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:39:54.392597 23434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/wal
uuid: "2585d75533f343b7a8e38be8e88d7cc1"
format_stamp: "Formatted at 2026-09-26 14:39:54 on dist-test-slave-x4sn"
server_key: "c84861d98a59d901d642f3aae488beaf"
server_key_iv: "9403a3ed8de0796aa4242cb2e0501919"
server_key_version: "encryptionkey@0"
I20260926 14:39:54.392702 23434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:54.421680 23434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:54.424889 23434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:54.425091 23434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:54.432029 23434 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.252:45751
I20260926 14:39:54.432301 23587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.252:45751 every 8 connection(s)
I20260926 14:39:54.432984 23434 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
I20260926 14:39:54.435714 23588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:54.439316 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 23434
I20260926 14:39:54.439445 23588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.252@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.90.252@KRBTEST.COM
I20260926 14:39:54.451171 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.441485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.16.90.254:44165)
0926 14:39:54.441607 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.441611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.441851 (+ 240us) server_negotiation.cc:407] Connection header received
0926 14:39:54.442309 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.442312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.442363 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.442439 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.443316 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.443876 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.444900 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.445135 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.445209 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.447710 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.447714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.447717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.447744 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.449183 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.449738 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.449740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.449741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.449807 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.450109 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.450112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.450114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.450289 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.450371 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.450894 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.451011 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:39:54.452104 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.440755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.16.90.252:45751)
0926 14:39:54.441180 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.441198 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.441237 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:39:54.442105 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.442124 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.442310 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.442460 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.443144 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.444144 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.444926 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.445310 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.445414 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.448154 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.448160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.448169 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.448189 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.449797 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.450157 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.450163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.450165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.450233 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.450482 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.450485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.450487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.450704 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.450889 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.451091 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.451220 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260926 14:39:54.452102 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.441346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:51698)
0926 14:39:54.441727 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.441765 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.442101 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.442503 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.442508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.442910 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.443181 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.443194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.443984 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.443987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.444792 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.444800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.445258 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.445262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.445411 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.446110 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.446125 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.447541 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.449281 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.449287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.449297 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.449656 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.449891 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.449894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.449896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.450025 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.450387 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.450394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.450628 (+ 234us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.450789 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.451100 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":6,"spinlock_wait_cycles":1920,"thread_start_us":140,"threads_started":1}
I20260926 14:39:54.453301 23588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.252@KRBTEST.COM for oryx/127.16.90.253@KRBTEST.COM
I20260926 14:39:54.461004 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.453765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:38709 (local address 127.0.0.1:52868)
0926 14:39:54.453991 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.454004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.454072 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.454299 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.454302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.454437 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.454635 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.454641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.455609 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.455612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.456196 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.456204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.456626 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.456629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.456786 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.457145 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.457155 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.458290 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.459757 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.459760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.459763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.459935 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.460205 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.460208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.460209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.460291 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.460601 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.460603 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.460682 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.460756 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.460852 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260926 14:39:54.461058 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.453868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52868 (local address 127.16.90.253:38709)
0926 14:39:54.454109 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.454113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.454126 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:39:54.454163 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.454166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.454207 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.454278 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.454750 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.455512 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.456296 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.456536 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.456622 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.458411 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.458414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.458417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.458441 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.459652 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.460014 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.460018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.460020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.460095 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.460373 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.460375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.460376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.460534 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.460606 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.460798 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.460912 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260926 14:39:54.461740 23588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.252@KRBTEST.COM for oryx/127.16.90.252@KRBTEST.COM
I20260926 14:39:54.468631 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.462179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:45751 (local address 127.0.0.1:43926)
0926 14:39:54.462282 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.462296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.462363 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.462553 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.462555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.462689 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.462755 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.462760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.463661 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.463663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.464186 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.464194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.464738 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.464741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.464793 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.465195 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.465205 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.466292 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.467528 (+ 1236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.467530 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.467531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.467754 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.467951 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.467953 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.467955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.468024 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.468271 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.468273 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.468335 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.468407 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.468501 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 14:39:54.468660 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.462221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.16.90.252:45751)
0926 14:39:54.462312 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.462315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.462330 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:39:54.462431 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.462434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.462471 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.462546 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.462845 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.463553 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.464286 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.464629 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.464737 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.466399 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.466402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.466404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.466432 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.467441 (+ 1009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.467816 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.467819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.467820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.467882 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.468074 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.468076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.468077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.468213 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.468276 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.468432 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.468550 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":38}
I20260926 14:39:54.471653 23588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: Bootstrap starting.
I20260926 14:39:54.473863 23588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:54.474536 23588 log.cc:824] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:54.476447 23588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: No bootstrap required, opened a new log
I20260926 14:39:54.479406 23588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } }
I20260926 14:39:54.479606 23588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:54.479648 23588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2585d75533f343b7a8e38be8e88d7cc1, State: Initialized, Role: FOLLOWER
I20260926 14:39:54.480123 23588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } }
I20260926 14:39:54.480631 23594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } } }
I20260926 14:39:54.480768 23594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:39:54.481302 23588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:54.484535 23053 heartbeater.cc:503] Master 127.16.90.254:44165 was elected leader, sending a full tablet report...
W20260926 14:39:54.486007 23605 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:39:54.486085 23605 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:39:54.489424 23434 master_runner.cc:428] Detected that this master 2585d75533f343b7a8e38be8e88d7cc1 is joining an existing cluster
I20260926 14:39:54.489471 23434 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.90.252:45751
I20260926 14:39:54.498741 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.491445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.16.90.254:44165)
0926 14:39:54.491655 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.491660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.492224 (+ 564us) server_negotiation.cc:407] Connection header received
0926 14:39:54.492367 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.492370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.492413 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.492460 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.493051 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.493556 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.494337 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.494552 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.494645 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.495688 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.495692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.495695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.495720 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.497093 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.497652 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.497655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.497657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.497741 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.498007 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.498011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.498012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.498178 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.498260 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.498492 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.498603 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260926 14:39:54.498744 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.491842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:51706)
0926 14:39:54.492146 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.492158 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.492275 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.492582 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.492584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.492823 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.492945 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.492951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.493656 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.493659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.494239 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.494246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.494655 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.494658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.494737 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.495181 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.495193 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.495541 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.497362 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.497364 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.497366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.497555 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.497850 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.497852 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.497855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.497927 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.498262 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.498266 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.498352 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.498434 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.498558 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":14,"thread_start_us":79,"threads_started":1}
I20260926 14:39:54.500310 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.491448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:38709 (local address 127.0.0.1:52874)
0926 14:39:54.491943 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.491958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.492110 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.492490 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.492493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.492844 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.493010 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.493017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.494069 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.494072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.494977 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.494987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.495758 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.495761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.495853 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.496347 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.496361 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.496776 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.498529 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.498532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.498535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.498757 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.499281 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.499285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.499287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.499398 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.499802 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.499806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.499895 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.500000 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.500135 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":15,"thread_start_us":95,"threads_started":1}
I20260926 14:39:54.500326 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.491843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.16.90.253:38709)
0926 14:39:54.491959 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.491963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.492022 (+ 59us) server_negotiation.cc:407] Connection header received
0926 14:39:54.492234 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.492237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.492294 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.492346 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.493154 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.493932 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.495125 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.495502 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.495577 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.496922 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.496925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.496928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.496953 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.498413 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.499069 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.499072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.499074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.499158 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.499486 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.499489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.499491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.499719 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.499808 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.500061 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.500190 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:39:54.505075 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 22923
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.254@KRBTEST.COM for oryx/127.16.90.252@KRBTEST.COM
I20260926 14:39:54.512185 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:39189
--local_ip_for_outbound_sockets=127.16.90.193
--tserver_master_addrs=127.16.90.254:44165
--webserver_port=41493
--webserver_interface=127.16.90.193
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab
--principal=oryx/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.90.254:44165,127.16.90.253:38709
--tserver_master_addrs=127.16.90.254:44165,127.16.90.253:38709,127.16.90.252:45751 with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:39:54.515208 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.502380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:45751 (local address 127.0.0.1:43946)
0926 14:39:54.502475 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.502488 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.502565 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.502922 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.502924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.503083 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.503165 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.503170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.503925 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.503928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.504637 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.504647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.505335 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.505338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.505420 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.505940 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.505953 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.507518 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.513609 (+ 6091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.513612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.513615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.513859 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.514188 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.514191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.514193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.514279 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.514693 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.514696 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.514794 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.514871 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.515021 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 14:39:54.515849 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.502450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.16.90.252:45751)
0926 14:39:54.502716 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.502721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.502731 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:39:54.502779 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.502781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.502818 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.502891 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.503263 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.503825 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.504765 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.505217 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.505841 (+ 624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.507719 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.507723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.507726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.507749 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.513477 (+ 5728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.513974 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.513977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.513979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.514056 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.514379 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.514382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.514383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.514578 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.515394 (+ 816us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.515469 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.515582 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":109,"threads_started":1}
I20260926 14:39:54.516870 22861 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.90.252:45751: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1"
member_type: NON_VOTER
last_known_addr {
host: "127.16.90.252"
port: 45751
}
attrs {
promote: true
}
}
dest_uuid: "4bc24156c7744341835e117b92b82b36"
cas_config_opid_index: 5
I20260926 14:39:54.517498 22861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } }
I20260926 14:39:54.518498 23423 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2585d75533f343b7a8e38be8e88d7cc1 (127.16.90.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:54.519003 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.491469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:45751 (local address 127.0.0.1:43936)
0926 14:39:54.492087 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.492100 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.492216 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.492507 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.492510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.511899 (+ 19389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.512022 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.512030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.512900 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.512903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.513644 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.513655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.514246 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.514249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.514330 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.514767 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.514780 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.515186 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.517344 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.517347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.517348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.517513 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.517766 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.517769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.517770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.517848 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.518516 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.518519 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.518608 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.518723 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.518845 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":93,"threads_started":1}
I20260926 14:39:54.519608 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.491536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.16.90.252:45751)
0926 14:39:54.491760 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.491764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.492162 (+ 398us) server_negotiation.cc:407] Connection header received
0926 14:39:54.492298 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.492301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.492336 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.492378 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.512154 (+ 19776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.512794 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.513759 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.514049 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.514143 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.515575 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.515578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.515580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.515600 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.517254 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.517592 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.517596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.517598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.517683 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.517917 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.517920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.517922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.518102 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.518182 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.519392 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.519490 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260926 14:39:54.519639 23562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:54.519985 23423 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.90.252:45751
I20260926 14:39:54.520169 23423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:54.520268 23423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:54.520601 23434 master.cc:537] Master@127.16.90.252:45751 shutting down...
I20260926 14:39:54.521054 23562 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 1 FOLLOWER]: Refusing update from remote peer 4bc24156c7744341835e117b92b82b36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 14:39:54.521641 23424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:39:54.523289 22848 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 14:39:54.525170 22848 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 -> Peer 2585d75533f343b7a8e38be8e88d7cc1 (127.16.90.252:45751): Couldn't send request to peer 2585d75533f343b7a8e38be8e88d7cc1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 14:39:54.526098 23434 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:54.526387 23434 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:54.526487 23434 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: stopping tablet replica
I20260926 14:39:54.528473 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.518569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:38709 (local address 127.0.0.1:52890)
0926 14:39:54.523251 (+ 4682us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.523265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.523337 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.523861 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.523863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.523880 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:54.523963 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.523970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.524957 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.524961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.526248 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.526259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.527874 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.527878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.527967 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.528012 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.528083 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.528174 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4606}
I20260926 14:39:54.528863 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.518822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.16.90.253:38709)
0926 14:39:54.523571 (+ 4749us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.523575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.523591 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:39:54.523651 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.523654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.523734 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.523825 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:54.524123 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.524832 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.526918 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.527522 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.527633 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.527811 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.528717 (+ 906us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.528780 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4658}
I20260926 14:39:54.529294 23480 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 LEARNER]: Refusing update from remote peer 4bc24156c7744341835e117b92b82b36: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 14:39:54.529726 23424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 14:39:54.530828 23480 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2585d75533f343b7a8e38be8e88d7cc1 (127.16.90.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:54.533154 23434 master.cc:552] Master@127.16.90.252:45751 shutdown complete.
I20260926 14:39:54.533038 23509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bc24156c7744341835e117b92b82b36. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:54.533195 23434 master_runner.cc:305] Clearing existing system tablet
I20260926 14:39:54.533200 23509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:39:54.533290 23509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: NON_VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: true } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:54.533358 23509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:39:54.533442 23434 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:54.533488 23434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:54.534431 23422 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36: attempting to promote NON_VOTER 16b721197e224140a9783e64705f54b6 to VOTER
I20260926 14:39:54.534727 23434 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:39:54.534896 23422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } }
I20260926 14:39:54.535454 23620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:54.535984 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.535733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:45751 (local address 127.0.0.1:43956)
0926 14:39:54.535838 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.535913 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:39:54.535946 23434 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:54.536234 23434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:54.536319 23480 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 LEARNER]: Refusing update from remote peer 4bc24156c7744341835e117b92b82b36: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 14:39:54.536322 23434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/wal
uuid: "2585d75533f343b7a8e38be8e88d7cc1"
format_stamp: "Formatted at 2026-09-26 14:39:54 on dist-test-slave-x4sn"
server_key: "c84861d98a59d901d642f3aae488beaf"
server_key_iv: "9403a3ed8de0796aa4242cb2e0501919"
server_key_version: "encryptionkey@0"
I20260926 14:39:54.536688 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.536439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:45751 (local address 127.0.0.1:43960)
0926 14:39:54.536570 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.536616 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 14:39:54.536695 23424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 14:39:54.537887 22848 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 -> Peer 2585d75533f343b7a8e38be8e88d7cc1 (127.16.90.252:45751): Couldn't send request to peer 2585d75533f343b7a8e38be8e88d7cc1. Status: Network error: Client connection negotiation failed: client connection to 127.16.90.252:45751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 14:39:54.538478 23424 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 16b721197e224140a9783e64705f54b6 (127.16.90.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:54.538772 23434 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:54.539912 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:54.540021 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:54.540156 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:54.540256 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:54.540879 23480 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 16b721197e224140a9783e64705f54b6 (127.16.90.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:54.541381 23434 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 14:39:54.541550 23434 log.cc:1196] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 14:39:54.541887 23434 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: Deleting consensus metadata
I20260926 14:39:54.542092 23434 master_runner.cc:315] Copying system tablet from 127.16.90.254:44165
I20260926 14:39:54.542394 23509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:54.542497 23509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:39:54.543329 23434 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.90.254:44165
I20260926 14:39:54.550170 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.543843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.16.90.254:44165)
0926 14:39:54.543942 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.543946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.544199 (+ 253us) server_negotiation.cc:407] Connection header received
0926 14:39:54.544306 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.544309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.544351 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.544413 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.544826 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.545328 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.546098 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.546324 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.546405 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.547402 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.547406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.547409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.547432 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.548754 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.549123 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.549126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.549128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.549200 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.549433 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.549436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.549438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.549604 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.549691 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.549928 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.550029 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:39:54.550189 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.543807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:51712)
0926 14:39:54.544113 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.544131 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.544226 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.544439 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.544442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.544630 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.544725 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.544731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.545424 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.545427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.545986 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.545994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.546417 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.546420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.546505 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.546958 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.546970 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.547280 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.548850 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.548853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.548856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.549041 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.549278 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.549281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.549283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.549359 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.549694 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.549697 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.549788 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.549869 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.550001 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":92,"threads_started":1}
I20260926 14:39:54.550700 22881 tablet_copy_service.cc:140] P 4bc24156c7744341835e117b92b82b36: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2585d75533f343b7a8e38be8e88d7cc1 ({username='oryx', principal='oryx/127.16.90.252@KRBTEST.COM'} at 127.0.0.1:51712)
I20260926 14:39:54.550804 22881 tablet_copy_service.cc:161] P 4bc24156c7744341835e117b92b82b36: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2585d75533f343b7a8e38be8e88d7cc1 at {username='oryx', principal='oryx/127.16.90.252@KRBTEST.COM'} at 127.0.0.1:51712: session id = 2585d75533f343b7a8e38be8e88d7cc1-00000000000000000000000000000000
I20260926 14:39:54.551898 22881 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 14:39:54.553143 23434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 14:39:54.555078 23434 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: tablet copy: Starting download of 0 data blocks...
I20260926 14:39:54.555244 23434 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: tablet copy: Starting download of 1 WAL segments...
I20260926 14:39:54.556558 23434 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 14:39:54.557936 22881 tablet_copy_service.cc:342] P 4bc24156c7744341835e117b92b82b36: Request end of tablet copy session 2585d75533f343b7a8e38be8e88d7cc1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.90.252@KRBTEST.COM'} at 127.0.0.1:51712
I20260926 14:39:54.558009 22881 tablet_copy_service.cc:433] P 4bc24156c7744341835e117b92b82b36: ending tablet copy session 2585d75533f343b7a8e38be8e88d7cc1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2585d75533f343b7a8e38be8e88d7cc1
I20260926 14:39:54.564788 23434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:54.566319 23628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:54.566413 23629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:54.566689 23631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:54.566845 23434 server_base.cc:1034] running on GCE node
I20260926 14:39:54.566977 23434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:54.567235 23434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:54.568359 23434 hybrid_clock.cc:648] HybridClock initialized: now 1790433594568352 us; error 23 us; skew 500 ppm
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:54.569669 23434 init.cc:377] Logged in from keytab as oryx/127.16.90.252@KRBTEST.COM (short username oryx)
I20260926 14:39:54.570139 23434 webserver.cc:460] Webserver started at http://127.16.90.252:40789/ using document root <none> and password file <none>
I20260926 14:39:54.570323 23434 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:54.570358 23434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:54.571487 23434 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:54.572361 23638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:54.572557 23434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:54.572621 23434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/wal
uuid: "2585d75533f343b7a8e38be8e88d7cc1"
format_stamp: "Formatted at 2026-09-26 14:39:54 on dist-test-slave-x4sn"
server_key: "c84861d98a59d901d642f3aae488beaf"
server_key_iv: "9403a3ed8de0796aa4242cb2e0501919"
server_key_version: "encryptionkey@0"
I20260926 14:39:54.572700 23434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:54.593998 23434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:54.595772 23434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:54.600194 23434 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.252:45751
I20260926 14:39:54.600494 23690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.252:45751 every 8 connection(s)
I20260926 14:39:54.600791 23434 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
I20260926 14:39:54.602173 23691 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:39:54.602965 23691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: Bootstrap starting.
I20260926 14:39:54.611369 23691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:39:54.611639 23691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: Bootstrap complete.
I20260926 14:39:54.612113 23691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } }
I20260926 14:39:54.612227 23691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2585d75533f343b7a8e38be8e88d7cc1, State: Initialized, Role: LEARNER
I20260926 14:39:54.612330 23691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } }
I20260926 14:39:54.612622 23694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:54.612787 23694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:39:54.612788 23691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:54.615818 23705 catalog_manager.cc:1259] Loaded cluster ID: bd408c3a2c0c4a139220ed59a17cd8ca
I20260926 14:39:54.615927 23705 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: loading cluster ID for follower catalog manager: success
I20260926 14:39:54.617600 23705 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: acquiring CA information for follower catalog manager: success
I20260926 14:39:54.618636 23705 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 14:39:54.678432 23615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:54.678817 23615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:54.678887 23615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:54.683019 23615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:54.683079 23615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:54.683139 23615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:54.688445 23615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab
--principal=oryx/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:39189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=41493
--tserver_master_addrs=127.16.90.254:44165,127.16.90.253:38709,127.16.90.252:45751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:54.689469 23615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:54.690438 23615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:39:54.693975 23188 heartbeater.cc:503] Master 127.16.90.254:44165 was elected leader, sending a full tablet report...
W20260926 14:39:54.699558 23711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:54.699571 23713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:54.699592 23710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:54.699865 23615 server_base.cc:1034] running on GCE node
I20260926 14:39:54.700241 23615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:54.700815 23615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:54.701960 23615 hybrid_clock.cc:648] HybridClock initialized: now 1790433594701946 us; error 31 us; skew 500 ppm
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:54.704550 23615 init.cc:377] Logged in from keytab as oryx/127.16.90.193@KRBTEST.COM (short username oryx)
I20260926 14:39:54.705461 23615 webserver.cc:460] Webserver started at http://127.16.90.193:41493/ using document root <none> and password file <none>
I20260926 14:39:54.705957 23615 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:54.706059 23615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:54.709729 23615 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:54.711602 23720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:54.712850 23615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:54.712944 23615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "6b35d833c2914abbb9412034f3541486"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "fddd7320e2f5923b6036f3901a69849f"
server_key_iv: "6d1437249b3c33df5c2f28664a218302"
server_key_version: "encryptionkey@0"
I20260926 14:39:54.713513 23615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:54.750067 23615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:54.753378 23615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:54.753552 23615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:54.754145 23615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:54.755102 23615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:54.755158 23615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:54.755203 23615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:54.755230 23615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:54.763623 23615 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:39189
I20260926 14:39:54.763686 23833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:39189 every 8 connection(s)
I20260926 14:39:54.764645 23615 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:39:54.765687 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 23615
I20260926 14:39:54.765872 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 23058
I20260926 14:39:54.772634 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:34647
--local_ip_for_outbound_sockets=127.16.90.194
--tserver_master_addrs=127.16.90.254:44165
--webserver_port=38091
--webserver_interface=127.16.90.194
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab
--principal=oryx/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.90.254:44165,127.16.90.253:38709
--tserver_master_addrs=127.16.90.254:44165,127.16.90.253:38709,127.16.90.252:45751 with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.253@KRBTEST.COM
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.252@KRBTEST.COM
I20260926 14:39:54.783917 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.767536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:40551 (local address 127.16.90.253:38709)
0926 14:39:54.767661 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.767685 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.770114 (+ 2429us) server_negotiation.cc:407] Connection header received
0926 14:39:54.770288 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.770292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.770345 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.770397 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.773237 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.774027 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.775206 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.775503 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.775619 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.779293 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.779297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.779300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.779329 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.781730 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.782193 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.782197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.782199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.782281 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.782553 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.782556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.782558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.782773 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.782847 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.783606 (+ 759us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.783755 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:39:54.783943 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.766960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:41719 (local address 127.16.90.254:44165)
0926 14:39:54.767323 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.767328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.768987 (+ 1659us) server_negotiation.cc:407] Connection header received
0926 14:39:54.770300 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.770304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.770360 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.770428 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.772327 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.773127 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.774847 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.775189 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.775285 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.779217 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.779222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.779225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.779252 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.781201 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.781860 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.781864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.781866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.781971 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.782437 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.782441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.782443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.782676 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.782762 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.783631 (+ 869us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.783780 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
I20260926 14:39:54.784641 23838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.767086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:58673 (local address 127.16.90.252:45751)
0926 14:39:54.768793 (+ 1707us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.768798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.768966 (+ 168us) server_negotiation.cc:407] Connection header received
0926 14:39:54.770341 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.770345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.770398 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.770471 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.775513 (+ 5042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.776564 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.777604 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.778050 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.778165 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.780394 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.780398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.780401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.780430 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.781907 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.782322 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.782325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.782327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.782407 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.782668 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.782671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.782673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.782862 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.782936 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.784353 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.784479 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1636,"thread_start_us":90,"threads_started":1}
I20260926 14:39:54.784976 23841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.766967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:38709 (local address 127.16.90.193:40551)
0926 14:39:54.770028 (+ 3061us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.770041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.770181 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.770717 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.770727 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.772970 (+ 2243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.773088 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.773095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.774198 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.774201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.775089 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.775097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.775768 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.775772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.775875 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.776350 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.776378 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.779122 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.781860 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.781863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.781865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.782085 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.782380 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.782383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.782385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.782460 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.782903 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.782906 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.783322 (+ 416us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.784125 (+ 803us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.784272 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2971,"mutex_wait_us":1167,"spinlock_wait_cycles":1045120,"thread_start_us":585,"threads_started":1}
I20260926 14:39:54.785027 23839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.767929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.193:41719)
0926 14:39:54.768906 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.768917 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.769964 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.770717 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.770726 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.771321 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.772133 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.772165 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.773330 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.773334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.774712 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.774722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.775335 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.775339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.775531 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.776356 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.776374 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.779028 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.781340 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.781346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.781363 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.781731 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.782138 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.782141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.782143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.782324 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.782833 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.782841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.783322 (+ 481us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.784155 (+ 833us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.784858 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":677,"spinlock_wait_cycles":22912,"thread_start_us":574,"threads_started":1}
I20260926 14:39:54.785307 23840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.766967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:45751 (local address 127.16.90.193:58673)
0926 14:39:54.768793 (+ 1826us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.768854 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.769955 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.770717 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.770727 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.771321 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.774503 (+ 3182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.775388 (+ 885us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.776707 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.776710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.777474 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.777484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.778212 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.778216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.778309 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.778757 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.778772 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.780233 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.782011 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.782014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.782016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.782224 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.782497 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.782499 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.782501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.782573 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.782972 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.782975 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.783322 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.783531 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.783890 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1615,"mutex_wait_us":94,"spinlock_wait_cycles":25088,"thread_start_us":141,"threads_started":1}
I20260926 14:39:54.786291 23834 heartbeater.cc:347] Connected to a master server at 127.16.90.252:45751
I20260926 14:39:54.786285 23836 heartbeater.cc:347] Connected to a master server at 127.16.90.253:38709
I20260926 14:39:54.786530 23834 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:54.786568 23836 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:54.786288 23835 heartbeater.cc:347] Connected to a master server at 127.16.90.254:44165
I20260926 14:39:54.786984 23835 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:54.787081 23834 heartbeater.cc:511] Master 127.16.90.252:45751 requested a full tablet report, sending...
I20260926 14:39:54.787125 23835 heartbeater.cc:511] Master 127.16.90.254:44165 requested a full tablet report, sending...
I20260926 14:39:54.787604 23836 heartbeater.cc:511] Master 127.16.90.253:38709 requested a full tablet report, sending...
I20260926 14:39:54.788359 22861 ts_manager.cc:194] Re-registered known tserver with Master: 6b35d833c2914abbb9412034f3541486 (127.16.90.193:39189)
I20260926 14:39:54.788523 23655 ts_manager.cc:194] Registered new tserver with Master: 6b35d833c2914abbb9412034f3541486 (127.16.90.193:39189)
I20260926 14:39:54.788591 23470 ts_manager.cc:194] Registered new tserver with Master: 6b35d833c2914abbb9412034f3541486 (127.16.90.193:39189)
I20260926 14:39:54.789095 22861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:41719
W20260926 14:39:54.885946 23842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:54.886185 23842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:54.886229 23842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:54.890144 23842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:54.890192 23842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:54.890262 23842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:54.897218 23842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab
--principal=oryx/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:34647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=38091
--tserver_master_addrs=127.16.90.254:44165,127.16.90.253:38709,127.16.90.252:45751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:54.898238 23842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:54.899243 23842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:39:54.904497 23322 heartbeater.cc:503] Master 127.16.90.254:44165 was elected leader, sending a full tablet report...
W20260926 14:39:54.908453 23850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:54.908486 23848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:54.908463 23847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:54.908727 23842 server_base.cc:1034] running on GCE node
I20260926 14:39:54.909091 23842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:54.909677 23842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:54.910823 23842 hybrid_clock.cc:648] HybridClock initialized: now 1790433594910808 us; error 25 us; skew 500 ppm
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:54.913632 23842 init.cc:377] Logged in from keytab as oryx/127.16.90.194@KRBTEST.COM (short username oryx)
I20260926 14:39:54.914467 23842 webserver.cc:460] Webserver started at http://127.16.90.194:38091/ using document root <none> and password file <none>
I20260926 14:39:54.915015 23842 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:54.915095 23842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:54.918802 23842 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:39:54.920694 23857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:54.921854 23842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:39:54.921953 23842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "f22762d682134b62bbbcb5e765dbb000"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "4add087f48a1038d31ec8f3fcb4391f4"
server_key_iv: "c90274a941511f78527428c7b91e41d7"
server_key_version: "encryptionkey@0"
I20260926 14:39:54.922443 23842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:54.938436 23842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:54.941226 23842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:54.941390 23842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:54.941936 23842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:54.942729 23842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:54.942777 23842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:54.942827 23842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:54.942853 23842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:54.951840 23842 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:34647
I20260926 14:39:54.951877 23971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:34647 every 8 connection(s)
I20260926 14:39:54.952831 23842 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:39:54.959312 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 23842
I20260926 14:39:54.959474 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 23193
I20260926 14:39:54.965855 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:44307
--local_ip_for_outbound_sockets=127.16.90.195
--tserver_master_addrs=127.16.90.254:44165
--webserver_port=39317
--webserver_interface=127.16.90.195
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab
--principal=oryx/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.90.254:44165,127.16.90.253:38709
--tserver_master_addrs=127.16.90.254:44165,127.16.90.253:38709,127.16.90.252:45751 with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.252@KRBTEST.COM
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.253@KRBTEST.COM
Sep 26 14:39:54 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
I20260926 14:39:54.971098 23838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.954630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:33383 (local address 127.16.90.252:45751)
0926 14:39:54.954901 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.954905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.956095 (+ 1190us) server_negotiation.cc:407] Connection header received
0926 14:39:54.956773 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.956776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.956831 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.956879 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.958152 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.958966 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.963283 (+ 4317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.963695 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.963778 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.966987 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.966991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.966994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.967022 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.968698 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.969245 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.969248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.969250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.969329 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.969718 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.969722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.969724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.969939 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.970012 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.970780 (+ 768us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.970930 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260926 14:39:54.971692 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.954616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:49973 (local address 127.16.90.254:44165)
0926 14:39:54.954802 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.954807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.955661 (+ 854us) server_negotiation.cc:407] Connection header received
0926 14:39:54.956725 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.956728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.956786 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.956857 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.959108 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.959864 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.963287 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.963645 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.963746 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.968280 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.968284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.968287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.968318 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.969824 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.970274 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.970277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.970278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.970356 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.970714 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.970717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.970719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.970924 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.971008 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.971376 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.971495 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 14:39:54.971737 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.955979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:40689 (local address 127.16.90.253:38709)
0926 14:39:54.956096 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:54.956100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:54.956115 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:39:54.956756 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:54.956759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:54.956813 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:54.956866 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:54.958138 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.958966 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.963288 (+ 4322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.963645 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.963746 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.967545 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:54.967550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:54.967553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:54.967585 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:54.969593 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.970042 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.970046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.970047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.970144 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:54.970454 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:54.970457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:54.970459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:54.970678 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:54.970759 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:54.971475 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 14:39:54.971583 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:39:54.972813 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.955522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:38709 (local address 127.16.90.194:40689)
0926 14:39:54.955930 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.955943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.956555 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.957021 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.957023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.957491 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.958004 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.958013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.959150 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.959154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.963173 (+ 4019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.963180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.963940 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.963944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.964231 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.964971 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.964984 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.967133 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.969721 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.969724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.969727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.969944 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.970279 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.970282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.970284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.970362 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.970798 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.970801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.970885 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.971002 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.972296 (+ 1294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":328,"spinlock_wait_cycles":4736,"thread_start_us":82,"threads_started":1}
I20260926 14:39:54.972822 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.954976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.194:49973)
0926 14:39:54.955511 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.955565 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.956530 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.957016 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.957023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.957470 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.957992 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.958006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.960265 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.960269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.963140 (+ 2871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.963151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.963940 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.963944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.964228 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.964900 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.964915 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.968119 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.969959 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.969962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.969964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.970171 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.970520 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.970523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.970525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.970595 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.971103 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.971105 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.971189 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.971902 (+ 713us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.972051 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":1735,"spinlock_wait_cycles":26880,"thread_start_us":129,"threads_started":1}
I20260926 14:39:54.973078 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:54.954976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:45751 (local address 127.16.90.194:33383)
0926 14:39:54.955970 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:54.955980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:54.956555 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:54.957018 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:54.957025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:54.957491 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:54.958006 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.958013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.960334 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.960337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.963152 (+ 2815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:54.963162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:54.963942 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:54.963945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:54.964184 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:54.964859 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:54.964879 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:54.966800 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:54.968818 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.968824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.968840 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.969136 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.969427 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:54.969430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:54.969432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:54.969601 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:54.970090 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:54.970096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:54.970473 (+ 377us) client_negotiation.cc:769] Sending connection context
0926 14:39:54.971314 (+ 841us) client_negotiation.cc:240] Negotiation successful
0926 14:39:54.971626 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":1717,"spinlock_wait_cycles":1408128,"thread_start_us":114,"threads_started":1}
I20260926 14:39:54.974203 23973 heartbeater.cc:347] Connected to a master server at 127.16.90.254:44165
I20260926 14:39:54.974397 23972 heartbeater.cc:347] Connected to a master server at 127.16.90.252:45751
I20260926 14:39:54.974472 23974 heartbeater.cc:347] Connected to a master server at 127.16.90.253:38709
I20260926 14:39:54.974498 23972 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:54.974529 23974 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:54.975003 23972 heartbeater.cc:511] Master 127.16.90.252:45751 requested a full tablet report, sending...
I20260926 14:39:54.975000 23974 heartbeater.cc:511] Master 127.16.90.253:38709 requested a full tablet report, sending...
I20260926 14:39:54.974429 23973 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:54.975711 23973 heartbeater.cc:511] Master 127.16.90.254:44165 requested a full tablet report, sending...
I20260926 14:39:54.976292 22861 ts_manager.cc:194] Re-registered known tserver with Master: f22762d682134b62bbbcb5e765dbb000 (127.16.90.194:34647)
I20260926 14:39:54.976375 23655 ts_manager.cc:194] Registered new tserver with Master: f22762d682134b62bbbcb5e765dbb000 (127.16.90.194:34647)
I20260926 14:39:54.976850 22861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:49973
I20260926 14:39:54.976965 23470 ts_manager.cc:194] Registered new tserver with Master: f22762d682134b62bbbcb5e765dbb000 (127.16.90.194:34647)
I20260926 14:39:55.022846 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.019220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:45751 (local address 127.0.0.1:43976)
0926 14:39:55.019440 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:55.019453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:55.019536 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:55.019857 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:55.019859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:55.019877 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:55.019963 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.019968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.020942 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.020944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.021683 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.021693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.022527 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.022530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.022598 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.022617 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:39:55.022670 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 14:39:55.022730 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260926 14:39:55.022907 23838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.019320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.16.90.252:45751)
0926 14:39:55.019470 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.019475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.019495 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:39:55.019637 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.019640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.019737 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.019840 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:55.020059 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.020841 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.021805 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.022425 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.022531 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.022689 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.022724 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.022783 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:39:55.023469 23665 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 14:39:55.023859 23694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bc24156c7744341835e117b92b82b36. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:55.023998 23694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: This master's current role is: LEARNER
W20260926 14:39:55.078169 23979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:55.078445 23979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:55.078490 23979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:55.082553 23979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:55.082612 23979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:55.082682 23979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:55.087819 23979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab
--principal=oryx/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:44307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=39317
--tserver_master_addrs=127.16.90.254:44165,127.16.90.253:38709,127.16.90.252:45751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:55.088836 23979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:55.089826 23979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:55.098919 23985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:55.098908 23987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:55.098884 23984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:55.099275 23979 server_base.cc:1034] running on GCE node
I20260926 14:39:55.099637 23979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:55.100262 23979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:55.101424 23979 hybrid_clock.cc:648] HybridClock initialized: now 1790433595101390 us; error 47 us; skew 500 ppm
Sep 26 14:39:55 dist-test-slave-x4sn krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433595, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:55.103972 23979 init.cc:377] Logged in from keytab as oryx/127.16.90.195@KRBTEST.COM (short username oryx)
I20260926 14:39:55.104808 23979 webserver.cc:460] Webserver started at http://127.16.90.195:39317/ using document root <none> and password file <none>
I20260926 14:39:55.105350 23979 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:55.105430 23979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:55.109162 23979 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260926 14:39:55.111093 23994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:55.112308 23979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:39:55.112406 23979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "603b1dae66d04c5baa64e503a0ae74d1"
format_stamp: "Formatted at 2026-09-26 14:39:53 on dist-test-slave-x4sn"
server_key: "414cb20a53d9acb2241d8aa2a8d13639"
server_key_iv: "31713f4e66f43db092411412d1822c07"
server_key_version: "encryptionkey@0"
I20260926 14:39:55.112932 23979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:55.130708 23979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:55.133522 23979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:55.133697 23979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:55.134264 23979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:39:55.135104 23979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:55.135147 23979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:55.135193 23979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:55.135213 23979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:55.143420 23979 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:44307
I20260926 14:39:55.143451 24108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:44307 every 8 connection(s)
I20260926 14:39:55.144393 23979 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:55.152608 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 23979
Sep 26 14:39:55 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433595, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
Sep 26 14:39:55 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433595, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.252@KRBTEST.COM
Sep 26 14:39:55 dist-test-slave-x4sn krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433595, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.253@KRBTEST.COM
I20260926 14:39:55.161121 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.146303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:48593 (local address 127.16.90.254:44165)
0926 14:39:55.146506 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.146510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.147930 (+ 1420us) server_negotiation.cc:407] Connection header received
0926 14:39:55.148517 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.148520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.148577 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.148650 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:55.150000 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.150821 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.152547 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.152892 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.152955 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.156848 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:55.156852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:55.156854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:55.156879 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:55.158436 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.159124 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.159128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.159130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.159214 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.159572 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.159575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.159577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.159819 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:55.159901 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.160846 (+ 945us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.160954 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260926 14:39:55.161808 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.146816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:45751 (local address 127.16.90.195:52897)
0926 14:39:55.148021 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:55.148030 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:55.148334 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:55.148849 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:55.148856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:55.149282 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:55.149883 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.149893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.150967 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.150970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.152415 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.152420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.153202 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.153205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.153298 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.153954 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:55.153966 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:55.156360 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:55.158785 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:55.158788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:55.158790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:55.159062 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:55.159362 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:55.159365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:55.159367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:55.159477 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:55.159946 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:55.159952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:55.160272 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 14:39:55.160465 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 14:39:55.160799 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":396,"spinlock_wait_cycles":18944,"thread_start_us":110,"threads_started":1}
I20260926 14:39:55.161866 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.146816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.195:48593)
0926 14:39:55.147829 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:55.147840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:55.148328 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:55.148848 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:55.148855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:55.149278 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:55.149873 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.149886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.150967 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.150970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.152407 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.152418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.153103 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.153106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.153197 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.153810 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:55.153825 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:55.156700 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:55.158559 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:55.158565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:55.158601 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:55.159015 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:55.159305 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:55.159308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:55.159310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:55.159468 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:55.159960 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:55.159963 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:55.160274 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 14:39:55.160468 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 14:39:55.161645 (+ 1177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":1142,"spinlock_wait_cycles":88192,"thread_start_us":135,"threads_started":1}
I20260926 14:39:55.162206 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.146303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:35037 (local address 127.16.90.253:38709)
0926 14:39:55.146429 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.146433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.147464 (+ 1031us) server_negotiation.cc:407] Connection header received
0926 14:39:55.148541 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.148545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.148597 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.148647 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:55.150945 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.151426 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.152368 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.152661 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.152770 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.157386 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:55.157390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:55.157392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:55.157420 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:55.159073 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.159487 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.159490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.159491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.159565 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.159857 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.159861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.159863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.160049 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:55.160133 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.161968 (+ 1835us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.162071 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:39:55.162303 23838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.146848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:52897 (local address 127.16.90.252:45751)
0926 14:39:55.146948 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.146952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.148105 (+ 1153us) server_negotiation.cc:407] Connection header received
0926 14:39:55.148542 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.148545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.148597 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.148647 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:55.150038 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.150821 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.152547 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.152883 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.152954 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.156511 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:55.156514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:55.156517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:55.156545 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:55.158666 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.159172 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.159175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.159177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.159263 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.159572 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.159575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.159577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.159818 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:55.159898 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.162062 (+ 2164us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.162163 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:39:55.162488 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.146816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:38709 (local address 127.16.90.195:35037)
0926 14:39:55.147311 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:55.147363 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:55.148327 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:55.148849 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:55.148856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:55.149280 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:55.149886 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.149893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.151529 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.151532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.152241 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.152250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.152928 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.152933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.153125 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.154094 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:55.154110 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:55.157241 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:55.159172 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:55.159174 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:55.159176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:55.159389 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:55.159654 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:55.159657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:55.159659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:55.159758 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:55.160171 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:55.160174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:55.160272 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 14:39:55.160675 (+ 403us) client_negotiation.cc:240] Negotiation successful
0926 14:39:55.161848 (+ 1173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":1121,"spinlock_wait_cycles":14976,"thread_start_us":149,"threads_started":1}
I20260926 14:39:55.163189 24109 heartbeater.cc:347] Connected to a master server at 127.16.90.252:45751
I20260926 14:39:55.163417 24109 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:55.163488 24110 heartbeater.cc:347] Connected to a master server at 127.16.90.254:44165
I20260926 14:39:55.163580 24110 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:55.163661 24111 heartbeater.cc:347] Connected to a master server at 127.16.90.253:38709
I20260926 14:39:55.163775 24111 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:55.164027 24111 heartbeater.cc:511] Master 127.16.90.253:38709 requested a full tablet report, sending...
I20260926 14:39:55.164029 24109 heartbeater.cc:511] Master 127.16.90.252:45751 requested a full tablet report, sending...
I20260926 14:39:55.164027 24110 heartbeater.cc:511] Master 127.16.90.254:44165 requested a full tablet report, sending...
I20260926 14:39:55.165139 23470 ts_manager.cc:194] Registered new tserver with Master: 603b1dae66d04c5baa64e503a0ae74d1 (127.16.90.195:44307)
I20260926 14:39:55.165206 23655 ts_manager.cc:194] Registered new tserver with Master: 603b1dae66d04c5baa64e503a0ae74d1 (127.16.90.195:44307)
I20260926 14:39:55.166257 22861 ts_manager.cc:194] Re-registered known tserver with Master: 603b1dae66d04c5baa64e503a0ae74d1 (127.16.90.195:44307)
I20260926 14:39:55.166946 22861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:48593
I20260926 14:39:55.166993 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.154877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.16.90.253:38709)
0926 14:39:55.156169 (+ 1292us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.156173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.156186 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:39:55.156218 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.156220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.156264 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.157687 (+ 1423us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:55.157783 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.158544 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.159652 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.159985 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.160069 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.161602 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:55.161606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:55.161609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:55.161632 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:55.164899 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.165355 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.165358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.165360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.165446 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.165760 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.165763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.165765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.165969 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:55.166085 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.166735 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.166846 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213,"thread_start_us":51,"threads_started":1}
I20260926 14:39:55.172307 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.154805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.16.90.254:44165)
0926 14:39:55.155016 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.155020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.155443 (+ 423us) server_negotiation.cc:407] Connection header received
0926 14:39:55.155606 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.155610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.155659 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.155757 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:55.162751 (+ 6994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.163351 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.166840 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.167192 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.167259 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.169160 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:55.169164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:55.169167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:55.169200 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:55.170525 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.171107 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.171110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.171111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.171197 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.171466 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.171469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.171470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.171695 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:55.171822 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.172012 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.172138 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":60,"threads_started":1}
I20260926 14:39:55.172328 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.155382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.16.90.252:45751)
0926 14:39:55.155585 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.155589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.164277 (+ 8688us) server_negotiation.cc:407] Connection header received
0926 14:39:55.164497 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.164500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.164548 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.164607 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:55.165604 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.166343 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.167392 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.167751 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.167854 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.169144 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:55.169147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:55.169150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:55.169174 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:55.170602 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.171289 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.171292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.171293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.171356 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.171567 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.171570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.171572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.171799 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:55.171900 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.172086 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.172189 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260926 14:39:55.174796 22861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:39:55.177723 22861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:55.180219 23621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: NON_VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: true } } }
I20260926 14:39:55.180349 23621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:55.184331 23621 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36: attempting to promote NON_VOTER 2585d75533f343b7a8e38be8e88d7cc1 to VOTER
I20260926 14:39:55.184670 23621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } }
I20260926 14:39:55.185842 23664 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 1 LEARNER]: Refusing update from remote peer 4bc24156c7744341835e117b92b82b36: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 14:39:55.185901 23480 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 FOLLOWER]: Refusing update from remote peer 4bc24156c7744341835e117b92b82b36: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 14:39:55.186290 24128 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 14:39:55.186631 24128 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 14:39:55.188555 24128 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2585d75533f343b7a8e38be8e88d7cc1 (127.16.90.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } } }
I20260926 14:39:55.188830 23480 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2585d75533f343b7a8e38be8e88d7cc1 (127.16.90.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } } }
I20260926 14:39:55.189405 23665 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2585d75533f343b7a8e38be8e88d7cc1 (127.16.90.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } } }
I20260926 14:39:55.190057 24128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } } }
I20260926 14:39:55.190143 24128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:55.190251 24128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } } }
I20260926 14:39:55.190308 24128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:55.191001 24137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } } }
I20260926 14:39:55.191110 24137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:39:55.191128 23694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } } }
I20260926 14:39:55.191232 23694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:39:55.191992 24128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4bc24156c7744341835e117b92b82b36" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } } }
I20260926 14:39:55.192090 24128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:55.197945 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.192084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:34647 (local address 127.0.0.1:36466)
0926 14:39:55.192583 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:55.192599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:55.192676 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:55.193885 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:55.193888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:55.193905 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:55.194003 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.194009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.195352 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.195355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.196524 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.196534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.197615 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.197618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.197700 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.197728 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 14:39:55.197771 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 14:39:55.197829 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423}
I20260926 14:39:55.198614 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.192237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.16.90.194:34647)
0926 14:39:55.193510 (+ 1273us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.193519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.193537 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:39:55.193594 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.193598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.193757 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.193893 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:55.194127 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.195242 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.196674 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.197487 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.198202 (+ 715us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.198337 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.198384 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.198473 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":97,"threads_started":1}
I20260926 14:39:55.198781 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.193116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.16.90.193:39189)
0926 14:39:55.193450 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.193461 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.193483 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:39:55.193555 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.193561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.193770 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.194504 (+ 734us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:55.194870 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.196041 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.197441 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.198219 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.198319 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.198444 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.198552 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.198637 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":128,"threads_started":1}
I20260926 14:39:55.198974 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.192361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:39189 (local address 127.0.0.1:50798)
0926 14:39:55.193182 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:55.193195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:55.193284 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:55.194650 (+ 1366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:55.194653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:55.194667 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:55.194758 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.194763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.196155 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.196158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.197314 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.197324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.198336 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.198339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.198423 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.198445 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:39:55.198524 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 14:39:55.198586 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":83,"threads_started":1}
I20260926 14:39:55.199193 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.192635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:44307 (local address 127.0.0.1:55966)
0926 14:39:55.194113 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:55.194125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:55.194201 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:55.195220 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:55.195223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:55.195240 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:55.195322 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.195328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.196816 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.196819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.197899 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.197908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.198843 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.198846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.198933 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.198955 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:39:55.199002 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 14:39:55.199062 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1422,"thread_start_us":80,"threads_started":1}
I20260926 14:39:55.199764 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.192976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.16.90.195:44307)
0926 14:39:55.194823 (+ 1847us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.194833 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.194854 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:39:55.194918 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.194923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.195079 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.195458 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:55.195585 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.196702 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.197975 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.198706 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.199343 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.199475 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.199523 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.199604 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1747,"thread_start_us":79,"threads_started":1}
I20260926 14:39:55.201519 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 1f43108b6c7b4c7fa8fdea15433a55c1 (DEFAULT_TABLE table=test-table [id=ad87755cc47e4730866704ff39857412]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:55.201946 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 1f43108b6c7b4c7fa8fdea15433a55c1 (DEFAULT_TABLE table=test-table [id=ad87755cc47e4730866704ff39857412]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:55.202456 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f43108b6c7b4c7fa8fdea15433a55c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:55.202239 24042 tablet_service.cc:1467] Processing CreateTablet for tablet 1f43108b6c7b4c7fa8fdea15433a55c1 (DEFAULT_TABLE table=test-table [id=ad87755cc47e4730866704ff39857412]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:55.202976 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f43108b6c7b4c7fa8fdea15433a55c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:55.203053 24042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f43108b6c7b4c7fa8fdea15433a55c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:55.208845 24143 tablet_bootstrap.cc:492] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000: Bootstrap starting.
I20260926 14:39:55.210485 24144 tablet_bootstrap.cc:492] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1: Bootstrap starting.
I20260926 14:39:55.211037 24143 tablet_bootstrap.cc:654] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:55.211477 24145 tablet_bootstrap.cc:492] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486: Bootstrap starting.
I20260926 14:39:55.211951 24143 log.cc:824] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:55.213495 24145 tablet_bootstrap.cc:654] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:55.213814 24144 tablet_bootstrap.cc:654] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:55.213966 24143 tablet_bootstrap.cc:492] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000: No bootstrap required, opened a new log
I20260926 14:39:55.214146 24143 ts_tablet_manager.cc:1397] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:39:55.214295 24145 log.cc:824] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:55.214618 24144 log.cc:824] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:55.216144 24145 tablet_bootstrap.cc:492] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486: No bootstrap required, opened a new log
I20260926 14:39:55.216315 24145 ts_tablet_manager.cc:1397] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:39:55.216652 24144 tablet_bootstrap.cc:492] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1: No bootstrap required, opened a new log
I20260926 14:39:55.216861 24144 ts_tablet_manager.cc:1397] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 14:39:55.218489 24143 raft_consensus.cc:348] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } } peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } }
I20260926 14:39:55.218762 24143 raft_consensus.cc:374] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:55.218819 24143 raft_consensus.cc:729] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22762d682134b62bbbcb5e765dbb000, State: Initialized, Role: FOLLOWER
I20260926 14:39:55.219269 24143 consensus_queue.cc:260] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } } peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } }
I20260926 14:39:55.219945 23973 heartbeater.cc:503] Master 127.16.90.254:44165 was elected leader, sending a full tablet report...
I20260926 14:39:55.220268 24143 ts_tablet_manager.cc:1428] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:39:55.221113 24145 raft_consensus.cc:348] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } } peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } }
I20260926 14:39:55.221362 24145 raft_consensus.cc:374] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:55.221408 24145 raft_consensus.cc:729] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b35d833c2914abbb9412034f3541486, State: Initialized, Role: FOLLOWER
I20260926 14:39:55.221838 24145 consensus_queue.cc:260] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } } peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } }
I20260926 14:39:55.222342 24144 raft_consensus.cc:348] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } } peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } }
I20260926 14:39:55.222589 24144 raft_consensus.cc:374] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:55.222642 24144 raft_consensus.cc:729] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603b1dae66d04c5baa64e503a0ae74d1, State: Initialized, Role: FOLLOWER
I20260926 14:39:55.222741 24145 ts_tablet_manager.cc:1428] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:39:55.223233 24144 consensus_queue.cc:260] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } } peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } }
I20260926 14:39:55.224268 24144 ts_tablet_manager.cc:1428] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 14:39:55.224627 24110 heartbeater.cc:503] Master 127.16.90.254:44165 was elected leader, sending a full tablet report...
I20260926 14:39:55.226486 23835 heartbeater.cc:503] Master 127.16.90.254:44165 was elected leader, sending a full tablet report...
W20260926 14:39:55.270010 23837 tablet.cc:2369] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:39:55.395850 24112 tablet.cc:2369] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:39:55.454169 23975 tablet.cc:2369] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:39:55.628664 24149 raft_consensus.cc:482] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:55.628862 24149 raft_consensus.cc:504] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } } peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } }
I20260926 14:39:55.629920 24149 leader_election.cc:288] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b35d833c2914abbb9412034f3541486 (127.16.90.193:39189), 603b1dae66d04c5baa64e503a0ae74d1 (127.16.90.195:44307)
I20260926 14:39:55.633838 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.630641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:43807 (local address 127.16.90.195:44307)
0926 14:39:55.630769 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.630773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.630814 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:39:55.630945 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.630948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.630989 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.631057 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:55.631275 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.631805 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.632910 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.633288 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.633379 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.633432 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.633649 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.633718 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:39:55.633858 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.630277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:36109 (local address 127.16.90.193:39189)
0926 14:39:55.630396 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.630399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.630474 (+ 75us) server_negotiation.cc:407] Connection header received
0926 14:39:55.630673 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.630688 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.630743 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.630796 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:55.631201 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.631785 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.632916 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.633312 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.633419 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.633469 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.633634 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.633717 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:39:55.633859 24152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.630153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:39189 (local address 127.16.90.194:36109)
0926 14:39:55.630397 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:55.630411 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:55.630521 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:55.630943 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:55.630945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:55.630965 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:55.631064 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.631070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.631886 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.631889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.632792 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.632801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.633420 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.633423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.633522 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.633545 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:39:55.633601 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:39:55.633668 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":48,"thread_start_us":84,"threads_started":1}
I20260926 14:39:55.634085 24151 raft_consensus.cc:482] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:55.634270 24151 raft_consensus.cc:504] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } } peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } }
I20260926 14:39:55.634632 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43108b6c7b4c7fa8fdea15433a55c1" candidate_uuid: "f22762d682134b62bbbcb5e765dbb000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b35d833c2914abbb9412034f3541486" is_pre_election: true
I20260926 14:39:55.634930 23788 raft_consensus.cc:2454] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f22762d682134b62bbbcb5e765dbb000 in term 0.
I20260926 14:39:55.633846 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.630526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:44307 (local address 127.16.90.194:43807)
0926 14:39:55.630743 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:55.630757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:55.630849 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:55.631069 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:55.631071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:55.631082 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:55.631176 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.631182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.631925 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.631928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.632789 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.632798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.633399 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.633403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.633531 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.633549 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:39:55.633623 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 14:39:55.633692 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":55,"thread_start_us":61,"threads_started":1}
I20260926 14:39:55.635376 23860 leader_election.cc:302] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b35d833c2914abbb9412034f3541486, f22762d682134b62bbbcb5e765dbb000; no voters:
I20260926 14:39:55.635528 24151 leader_election.cc:288] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f22762d682134b62bbbcb5e765dbb000 (127.16.90.194:34647), 6b35d833c2914abbb9412034f3541486 (127.16.90.193:39189)
I20260926 14:39:55.635629 24149 raft_consensus.cc:2781] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:55.635743 24149 raft_consensus.cc:482] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:55.635792 24149 raft_consensus.cc:3037] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:55.636018 24063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43108b6c7b4c7fa8fdea15433a55c1" candidate_uuid: "f22762d682134b62bbbcb5e765dbb000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603b1dae66d04c5baa64e503a0ae74d1" is_pre_election: true
I20260926 14:39:55.636198 24063 raft_consensus.cc:2454] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f22762d682134b62bbbcb5e765dbb000 in term 0.
I20260926 14:39:55.637040 24149 raft_consensus.cc:504] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } } peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } }
I20260926 14:39:55.637347 24149 leader_election.cc:288] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [CANDIDATE]: Term 1 election: Requested vote from peers 6b35d833c2914abbb9412034f3541486 (127.16.90.193:39189), 603b1dae66d04c5baa64e503a0ae74d1 (127.16.90.195:44307)
I20260926 14:39:55.638111 24063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43108b6c7b4c7fa8fdea15433a55c1" candidate_uuid: "f22762d682134b62bbbcb5e765dbb000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603b1dae66d04c5baa64e503a0ae74d1"
I20260926 14:39:55.638236 24063 raft_consensus.cc:3037] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:55.638962 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43108b6c7b4c7fa8fdea15433a55c1" candidate_uuid: "f22762d682134b62bbbcb5e765dbb000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b35d833c2914abbb9412034f3541486"
I20260926 14:39:55.639081 23788 raft_consensus.cc:3037] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:55.639421 24063 raft_consensus.cc:2454] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f22762d682134b62bbbcb5e765dbb000 in term 1.
I20260926 14:39:55.639715 23860 leader_election.cc:302] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603b1dae66d04c5baa64e503a0ae74d1, f22762d682134b62bbbcb5e765dbb000; no voters:
I20260926 14:39:55.639869 24149 raft_consensus.cc:2781] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:55.640074 24149 raft_consensus.cc:686] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 1 LEADER]: Becoming Leader. State: Replica: f22762d682134b62bbbcb5e765dbb000, State: Running, Role: LEADER
I20260926 14:39:55.640101 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.635713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:34647 (local address 127.16.90.195:53259)
0926 14:39:55.635813 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:55.635828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:55.635900 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:55.636989 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:55.636991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:55.637007 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:55.637104 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.637110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.638015 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.638019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.638821 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.638830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.639717 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.639721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.639824 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.639847 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:39:55.639916 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:39:55.639985 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":1}
I20260926 14:39:55.640301 23788 raft_consensus.cc:2454] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f22762d682134b62bbbcb5e765dbb000 in term 1.
I20260926 14:39:55.640375 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.636480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:53259 (local address 127.16.90.194:34647)
0926 14:39:55.636574 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.636578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.636589 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:39:55.636625 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.636628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.636665 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.636739 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:55.637242 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.637918 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.638952 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.639500 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.639586 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.639632 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.640221 (+ 589us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.640284 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:39:55.640403 24149 consensus_queue.cc:237] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } } peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } }
I20260926 14:39:55.641005 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.635955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:39189 (local address 127.16.90.195:48305)
0926 14:39:55.636036 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:55.636047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:55.636099 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:55.636754 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:55.636756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:55.636774 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:55.636870 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.636876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.637699 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.637702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.638774 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:55.638783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.640636 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.640640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:55.640748 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.640771 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:39:55.640837 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:39:55.640899 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":2}
I20260926 14:39:55.641115 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43108b6c7b4c7fa8fdea15433a55c1" candidate_uuid: "603b1dae66d04c5baa64e503a0ae74d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f22762d682134b62bbbcb5e765dbb000" is_pre_election: true
I20260926 14:39:55.641304 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.636480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:48305 (local address 127.16.90.193:39189)
0926 14:39:55.636574 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.636578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.636590 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:39:55.636625 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.636628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.636676 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.636738 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:55.636968 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.637556 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.638896 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.639481 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.639587 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.639632 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.641138 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.641209 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:39:55.641578 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43108b6c7b4c7fa8fdea15433a55c1" candidate_uuid: "603b1dae66d04c5baa64e503a0ae74d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b35d833c2914abbb9412034f3541486" is_pre_election: true
I20260926 14:39:55.641738 23788 raft_consensus.cc:2379] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 603b1dae66d04c5baa64e503a0ae74d1 in current term 1: Already voted for candidate f22762d682134b62bbbcb5e765dbb000 in this term.
I20260926 14:39:55.642125 23997 leader_election.cc:302] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 603b1dae66d04c5baa64e503a0ae74d1; no voters: 6b35d833c2914abbb9412034f3541486, f22762d682134b62bbbcb5e765dbb000
I20260926 14:39:55.642515 24151 raft_consensus.cc:2726] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 14:39:55.644291 22861 catalog_manager.cc:5495] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 reported cstate change: term changed from 0 to 1, leader changed from <none> to f22762d682134b62bbbcb5e765dbb000 (127.16.90.194). New cstate: current_term: 1 leader_uuid: "f22762d682134b62bbbcb5e765dbb000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } health_report { overall_health: UNKNOWN } } }
I20260926 14:39:55.757835 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.754381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.16.90.194:34647)
0926 14:39:55.754510 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.754514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.754528 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:39:55.754704 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.754706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.754761 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.754853 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:55.755090 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.755618 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.756465 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.756677 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.756762 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.757048 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:55.757471 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:55.757574 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.757652 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.757704 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:39:55.763223 23788 raft_consensus.cc:1261] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486 [term 1 FOLLOWER]: Refusing update from remote peer f22762d682134b62bbbcb5e765dbb000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:55.763286 24063 raft_consensus.cc:1261] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [term 1 FOLLOWER]: Refusing update from remote peer f22762d682134b62bbbcb5e765dbb000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:55.763870 24149 consensus_queue.cc:1035] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:55.764058 24154 consensus_queue.cc:1035] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:55.771716 24161 mvcc.cc:205] Tried to move back new op lower bound from 7333616008239452160 to 7333616007746478080. Current Snapshot: MvccSnapshot[applied={T|T < 7333616008239452160}]
I20260926 14:39:55.775736 24158 mvcc.cc:205] Tried to move back new op lower bound from 7333616008239452160 to 7333616007746478080. Current Snapshot: MvccSnapshot[applied={T|T < 7333616008239452160}]
I20260926 14:39:55.780354 24160 mvcc.cc:205] Tried to move back new op lower bound from 7333616008239452160 to 7333616007746478080. Current Snapshot: MvccSnapshot[applied={T|T < 7333616008239452160}]
I20260926 14:39:55.790794 22861 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:55.790948 22861 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:55.796435 22861 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bc24156c7744341835e117b92b82b36: Sending DeleteTablet for 3 replicas of tablet 1f43108b6c7b4c7fa8fdea15433a55c1
I20260926 14:39:55.797149 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f43108b6c7b4c7fa8fdea15433a55c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:55 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:36466
I20260926 14:39:55.797474 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f43108b6c7b4c7fa8fdea15433a55c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:55 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:50798
I20260926 14:39:55.797699 24168 tablet_replica.cc:330] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000: stopping tablet replica
I20260926 14:39:55.797952 24042 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f43108b6c7b4c7fa8fdea15433a55c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:55 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:55966
I20260926 14:39:55.798035 24168 raft_consensus.cc:2229] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:39:55.798472 24168 raft_consensus.cc:2258] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:55.799543 24169 tablet_replica.cc:330] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486: stopping tablet replica
I20260926 14:39:55.800175 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 22828
I20260926 14:39:55.800200 24168 ts_tablet_manager.cc:1905] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:55.803074 24168 ts_tablet_manager.cc:1918] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:39:55.803256 24168 log.cc:1196] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/1f43108b6c7b4c7fa8fdea15433a55c1
I20260926 14:39:55.803637 24168 ts_tablet_manager.cc:1939] T 1f43108b6c7b4c7fa8fdea15433a55c1 P f22762d682134b62bbbcb5e765dbb000: Deleting consensus metadata
I20260926 14:39:55.799921 24169 raft_consensus.cc:2229] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:55.808190 24169 raft_consensus.cc:2258] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:39:55.808579 23859 connection.cc:535] server connection from 127.0.0.1:36466 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:39:55.808717 24170 tablet_replica.cc:330] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1: stopping tablet replica
I20260926 14:39:55.809096 24170 raft_consensus.cc:2229] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:55.809469 24170 raft_consensus.cc:2258] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:55.809724 24169 ts_tablet_manager.cc:1905] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:55.810510 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.798912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.16.90.253:38709)
0926 14:39:55.800117 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.800122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.800137 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:39:55.800181 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.800184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.800239 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.800869 (+ 630us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:55.801843 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.802643 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.803860 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.804214 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.804317 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.806342 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:55.806347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:55.806350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:55.806377 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:55.808150 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.808689 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.808693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.808695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.808791 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.809119 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.809122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.809124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.809360 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:55.809503 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.810188 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.810317 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":73,"threads_started":1}
I20260926 14:39:55.811062 24170 ts_tablet_manager.cc:1905] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:55.813694 24170 ts_tablet_manager.cc:1918] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:39:55.813838 24170 log.cc:1196] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/1f43108b6c7b4c7fa8fdea15433a55c1
I20260926 14:39:55.814231 24170 ts_tablet_manager.cc:1939] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 603b1dae66d04c5baa64e503a0ae74d1: Deleting consensus metadata
W20260926 14:39:55.815393 23997 connection.cc:441] server connection from 127.0.0.1:55966 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55966 (request call id 1) could send its response
I20260926 14:39:55.816490 24169 ts_tablet_manager.cc:1918] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:39:55.816624 24169 log.cc:1196] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-0/wal/wals/1f43108b6c7b4c7fa8fdea15433a55c1
I20260926 14:39:55.816958 24169 ts_tablet_manager.cc:1939] T 1f43108b6c7b4c7fa8fdea15433a55c1 P 6b35d833c2914abbb9412034f3541486: Deleting consensus metadata
W20260926 14:39:55.818041 23724 connection.cc:441] server connection from 127.0.0.1:50798 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50798 (request call id 1) could send its response
I20260926 14:39:55.821115 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.798793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.16.90.252:45751)
0926 14:39:55.805276 (+ 6483us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.805282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.810967 (+ 5685us) server_negotiation.cc:407] Connection header received
0926 14:39:55.811235 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.811239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.811299 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.811374 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:55.812304 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.813063 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.814145 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.814483 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.814571 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.816986 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:55.816991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:55.816994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:55.817024 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:55.818811 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.819230 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.819234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.819235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.819325 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.819646 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.819649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.819651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.819911 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:55.820067 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.820720 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.820857 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":6401,"thread_start_us":68,"threads_started":1}
I20260926 14:39:55.825897 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.814715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.16.90.253:38709)
0926 14:39:55.815231 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.815236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.816247 (+ 1011us) server_negotiation.cc:407] Connection header received
0926 14:39:55.816646 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.816650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.816700 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.816777 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:55.817587 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.818402 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.819561 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.819940 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.820033 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.822072 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:55.822077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:55.822079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:55.822104 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:55.824062 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.824548 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.824552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.824554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.824661 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.824948 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.824952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.824954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.825232 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:55.825403 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.825620 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.825733 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437}
I20260926 14:39:55.827400 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:55.813562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.16.90.252:45751)
0926 14:39:55.819315 (+ 5753us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:55.819319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:55.819333 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:39:55.819365 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:55.819368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:55.819411 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:55.819508 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:55.820390 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.821318 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.822338 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:55.822630 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:55.822726 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:55.824038 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:55.824042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:55.824045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:55.824071 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:55.825766 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.826215 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.826218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.826220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.826299 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:55.826567 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:55.826571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:55.826573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:55.826775 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:55.826895 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:55.827127 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 14:39:55.827240 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5681,"thread_start_us":80,"threads_started":1}
I20260926 14:39:56.779831 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:56.779256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.193:37023)
0926 14:39:56.779599 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:56.779729 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":77,"threads_started":1}
I20260926 14:39:56.779930 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:56.779430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.194:51849)
0926 14:39:56.779742 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:56.779831 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":67,"threads_started":1}
W20260926 14:39:56.780290 23835 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:44165 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
W20260926 14:39:56.780344 23973 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:44165 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
I20260926 14:39:56.785475 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:56.785097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.195:54257)
0926 14:39:56.785321 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:56.785394 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
W20260926 14:39:56.786269 24110 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:44165 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
I20260926 14:39:57.295920 24185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bc24156c7744341835e117b92b82b36)
I20260926 14:39:57.296061 24185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } }
I20260926 14:39:57.296557 24185 leader_election.cc:288] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4bc24156c7744341835e117b92b82b36 (127.16.90.254:44165), 2585d75533f343b7a8e38be8e88d7cc1 (127.16.90.252:45751)
I20260926 14:39:57.296850 24186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bc24156c7744341835e117b92b82b36)
I20260926 14:39:57.297042 24186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } }
I20260926 14:39:57.297288 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.296727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:52142)
0926 14:39:57.297046 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.297136 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":65,"threads_started":1}
W20260926 14:39:57.297487 23456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
I20260926 14:39:57.298157 24186 leader_election.cc:288] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4bc24156c7744341835e117b92b82b36 (127.16.90.254:44165), 16b721197e224140a9783e64705f54b6 (127.16.90.253:38709)
I20260926 14:39:57.298674 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.298249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:52156)
0926 14:39:57.298516 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.298586 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":53,"threads_started":1}
W20260926 14:39:57.298830 23641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
I20260926 14:39:57.298866 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.298068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:52154)
0926 14:39:57.298662 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.298711 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523}
W20260926 14:39:57.299014 23456 leader_election.cc:334] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4bc24156c7744341835e117b92b82b36 (127.16.90.254:44165): Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
I20260926 14:39:57.299731 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.299500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:52170)
0926 14:39:57.299598 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.299645 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 14:39:57.299857 23641 leader_election.cc:334] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4bc24156c7744341835e117b92b82b36 (127.16.90.254:44165): Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
I20260926 14:39:57.301391 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.296956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.16.90.252:45751)
0926 14:39:57.297211 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:57.297216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:57.297234 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:39:57.297450 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:57.297453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:57.297501 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:57.297582 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:57.297953 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.299088 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.300432 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.300910 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.300985 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:57.301029 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:57.301257 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 14:39:57.301311 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
I20260926 14:39:57.301416 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.296730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:45751 (local address 127.0.0.1:44030)
0926 14:39:57.297107 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.297120 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:57.297302 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:57.297645 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:57.297649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:57.297675 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:57.297782 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:57.297788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.299226 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.299229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:57.300315 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:57.300326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.301016 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.301020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:57.301107 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:57.301131 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:39:57.301216 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:39:57.301281 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":132,"thread_start_us":48,"threads_started":1}
I20260926 14:39:57.301836 23665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16b721197e224140a9783e64705f54b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2585d75533f343b7a8e38be8e88d7cc1" is_pre_election: true
I20260926 14:39:57.302027 23665 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16b721197e224140a9783e64705f54b6 in term 1.
I20260926 14:39:57.302356 23457 leader_election.cc:302] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16b721197e224140a9783e64705f54b6, 2585d75533f343b7a8e38be8e88d7cc1; no voters: 4bc24156c7744341835e117b92b82b36
I20260926 14:39:57.302559 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.298328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:38709 (local address 127.0.0.1:52904)
0926 14:39:57.298551 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.298563 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:57.298644 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:57.299080 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:57.299082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:57.299099 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:57.299183 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:57.299190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.300399 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.300402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:57.301418 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:57.301429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.302216 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.302219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:57.302305 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:57.302328 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:39:57.302383 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 14:39:57.302435 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":41,"thread_start_us":78,"threads_started":1}
I20260926 14:39:57.302580 24185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 14:39:57.302590 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.298440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.16.90.253:38709)
0926 14:39:57.298857 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:57.298866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:57.298879 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:39:57.298918 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:57.298920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:57.298959 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:57.299306 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:57.299708 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.300298 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.301546 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.302093 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.302188 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:57.302242 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:57.302409 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 14:39:57.302478 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":240,"server-negotiator.queue_time_us":354,"thread_start_us":82,"threads_started":1}
I20260926 14:39:57.302649 24185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bc24156c7744341835e117b92b82b36)
I20260926 14:39:57.302690 24185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:39:57.303000 23480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2585d75533f343b7a8e38be8e88d7cc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "16b721197e224140a9783e64705f54b6" is_pre_election: true
I20260926 14:39:57.303808 24185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } }
I20260926 14:39:57.304037 23480 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2585d75533f343b7a8e38be8e88d7cc1 in current term 2: Already voted for candidate 16b721197e224140a9783e64705f54b6 in this term.
I20260926 14:39:57.304145 24185 leader_election.cc:288] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [CANDIDATE]: Term 2 election: Requested vote from peers 4bc24156c7744341835e117b92b82b36 (127.16.90.254:44165), 2585d75533f343b7a8e38be8e88d7cc1 (127.16.90.252:45751)
I20260926 14:39:57.304425 23665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16b721197e224140a9783e64705f54b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2585d75533f343b7a8e38be8e88d7cc1"
I20260926 14:39:57.304471 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.304268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:52178)
0926 14:39:57.304362 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.304415 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 14:39:57.304512 23665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:39:57.304558 23640 leader_election.cc:302] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2585d75533f343b7a8e38be8e88d7cc1; no voters: 16b721197e224140a9783e64705f54b6, 4bc24156c7744341835e117b92b82b36
I20260926 14:39:57.305125 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.304947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:52184)
0926 14:39:57.305039 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.305081 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 14:39:57.305223 23456 leader_election.cc:334] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4bc24156c7744341835e117b92b82b36 (127.16.90.254:44165): Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
I20260926 14:39:57.305568 23665 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16b721197e224140a9783e64705f54b6 in term 2.
I20260926 14:39:57.305656 24186 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:39:57.305868 23457 leader_election.cc:302] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16b721197e224140a9783e64705f54b6, 2585d75533f343b7a8e38be8e88d7cc1; no voters: 4bc24156c7744341835e117b92b82b36
I20260926 14:39:57.306118 24185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:39:57.306314 24185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [term 2 LEADER]: Becoming Leader. State: Replica: 16b721197e224140a9783e64705f54b6, State: Running, Role: LEADER
I20260926 14:39:57.306546 24185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } }
I20260926 14:39:57.307665 24196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16b721197e224140a9783e64705f54b6. Latest consensus state: current_term: 2 leader_uuid: "16b721197e224140a9783e64705f54b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } } }
I20260926 14:39:57.307791 24196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:57.308054 24198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:57.310202 24198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:57.310583 24198 catalog_manager.cc:1259] Loaded cluster ID: bd408c3a2c0c4a139220ed59a17cd8ca
I20260926 14:39:57.310628 24198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:57.311136 24198 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:57.311525 24198 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: Loaded TSK: 0
I20260926 14:39:57.312217 24198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:39:57.330791 23469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:39:57.333804 23469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:57.336499 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.336290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:52202)
0926 14:39:57.336395 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.336440 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 14:39:57.336557 23665 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [term 2 FOLLOWER]: Refusing update from remote peer 16b721197e224140a9783e64705f54b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 14:39:57.336987 24196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 14:39:57.337105 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.336921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.0.0.1:52214)
0926 14:39:57.337012 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.337055 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:39:57.337320 23456 consensus_peers.cc:487] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 -> Peer 4bc24156c7744341835e117b92b82b36 (127.16.90.254:44165): Couldn't send request to peer 4bc24156c7744341835e117b92b82b36. Status: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 14:39:57.338375 24186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16b721197e224140a9783e64705f54b6. Latest consensus state: current_term: 2 leader_uuid: "16b721197e224140a9783e64705f54b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } } }
I20260926 14:39:57.338500 24186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:39:57.339438 24196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "16b721197e224140a9783e64705f54b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } } }
I20260926 14:39:57.339550 24196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6 [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:57.339906 24186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "16b721197e224140a9783e64705f54b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4bc24156c7744341835e117b92b82b36" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44165 } } peers { permanent_uuid: "16b721197e224140a9783e64705f54b6" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 38709 } attrs { promote: false } } peers { permanent_uuid: "2585d75533f343b7a8e38be8e88d7cc1" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 45751 } attrs { promote: false } } }
I20260926 14:39:57.339994 24186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2585d75533f343b7a8e38be8e88d7cc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:39:57.348472 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.344833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:34647 (local address 127.0.0.1:36486)
0926 14:39:57.345057 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.345070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:57.345142 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:57.345532 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:57.345535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:57.345553 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:57.345671 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:57.345677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.346549 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.346552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:57.347374 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:57.347382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.348090 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.348094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:57.348183 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:57.348207 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:39:57.348277 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 14:39:57.348347 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":53}
I20260926 14:39:57.348505 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.345068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.16.90.194:34647)
0926 14:39:57.345290 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:57.345294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:57.345309 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:39:57.345359 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:57.345362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:57.345410 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:57.345782 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:57.345894 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.346446 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.347486 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.347958 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.348092 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:57.348157 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:57.348298 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 14:39:57.348373 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260926 14:39:57.349120 23905 tablet_service.cc:1467] Processing CreateTablet for tablet da89bf5ba3504099b2260093cf0d528c (DEFAULT_TABLE table=test-table [id=31d81f1ab48b4580b062c11d6cba3ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:57.349249 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.344589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:44307 (local address 127.0.0.1:55972)
0926 14:39:57.344700 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.344714 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:57.344789 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:57.345630 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:57.345633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:57.345649 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:57.345727 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:57.345731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.346568 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.346571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:57.347734 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:57.347743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.348895 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.348898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:57.348978 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:57.349001 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:39:57.349065 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:39:57.349128 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":23}
I20260926 14:39:57.349406 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da89bf5ba3504099b2260093cf0d528c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:57.349592 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.344730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.16.90.195:44307)
0926 14:39:57.345392 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:57.345397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:57.345410 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:39:57.345457 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:57.345460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:57.345504 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:57.345828 (+ 324us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:57.345940 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.346453 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.347859 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.348484 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.348578 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:57.348640 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:57.349385 (+ 745us) server_negotiation.cc:299] Negotiation successful
0926 14:39:57.349468 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":86,"threads_started":1}
I20260926 14:39:57.350070 24042 tablet_service.cc:1467] Processing CreateTablet for tablet da89bf5ba3504099b2260093cf0d528c (DEFAULT_TABLE table=test-table [id=31d81f1ab48b4580b062c11d6cba3ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:57.350378 24042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da89bf5ba3504099b2260093cf0d528c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:57.350509 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.345355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:39189 (local address 127.0.0.1:50810)
0926 14:39:57.345865 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.345877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:57.345951 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:57.346970 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:57.346973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:57.346988 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:57.347062 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:57.347067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.348024 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.348027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:57.349043 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:57.349052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.350168 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.350171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:57.350249 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:57.350269 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:39:57.350329 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 14:39:57.350390 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":83,"threads_started":1}
I20260926 14:39:57.350960 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.345472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.16.90.193:39189)
0926 14:39:57.346691 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:57.346696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:57.346709 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:39:57.346755 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:57.346758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:57.346800 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:57.346875 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:57.347180 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.347912 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.349187 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.349960 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.350048 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:57.350105 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:57.350747 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 14:39:57.350830 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1132,"thread_start_us":71,"threads_started":1}
I20260926 14:39:57.351401 23768 tablet_service.cc:1467] Processing CreateTablet for tablet da89bf5ba3504099b2260093cf0d528c (DEFAULT_TABLE table=test-table [id=31d81f1ab48b4580b062c11d6cba3ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:57.351650 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da89bf5ba3504099b2260093cf0d528c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:57.352321 24209 tablet_bootstrap.cc:492] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000: Bootstrap starting.
I20260926 14:39:57.352828 24210 tablet_bootstrap.cc:492] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1: Bootstrap starting.
I20260926 14:39:57.353264 24209 tablet_bootstrap.cc:654] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:57.353662 24210 tablet_bootstrap.cc:654] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:57.354296 24209 tablet_bootstrap.cc:492] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000: No bootstrap required, opened a new log
I20260926 14:39:57.354380 24209 ts_tablet_manager.cc:1397] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:39:57.354665 24210 tablet_bootstrap.cc:492] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1: No bootstrap required, opened a new log
I20260926 14:39:57.354732 24210 ts_tablet_manager.cc:1397] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:39:57.354727 24209 raft_consensus.cc:348] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } } peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } }
I20260926 14:39:57.354836 24209 raft_consensus.cc:374] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:57.354871 24209 raft_consensus.cc:729] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22762d682134b62bbbcb5e765dbb000, State: Initialized, Role: FOLLOWER
I20260926 14:39:57.354990 24209 consensus_queue.cc:260] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } } peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } }
I20260926 14:39:57.355118 24210 raft_consensus.cc:348] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } } peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } }
I20260926 14:39:57.355228 24210 raft_consensus.cc:374] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:57.355266 24210 raft_consensus.cc:729] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603b1dae66d04c5baa64e503a0ae74d1, State: Initialized, Role: FOLLOWER
I20260926 14:39:57.355266 24209 ts_tablet_manager.cc:1428] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:39:57.355379 24210 consensus_queue.cc:260] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } } peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } }
I20260926 14:39:57.355523 24212 tablet_bootstrap.cc:492] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486: Bootstrap starting.
I20260926 14:39:57.356846 24212 tablet_bootstrap.cc:654] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:57.357583 24210 ts_tablet_manager.cc:1428] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 14:39:57.357759 23974 heartbeater.cc:503] Master 127.16.90.253:38709 was elected leader, sending a full tablet report...
I20260926 14:39:57.358487 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.358125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.195:53427)
0926 14:39:57.358340 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.358397 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260926 14:39:57.358552 24215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.355716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.194:42243)
0926 14:39:57.358405 (+ 2689us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.358463 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2625,"thread_start_us":87,"threads_started":1}
I20260926 14:39:57.359630 24212 tablet_bootstrap.cc:492] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486: No bootstrap required, opened a new log
I20260926 14:39:57.359715 24111 heartbeater.cc:503] Master 127.16.90.253:38709 was elected leader, sending a full tablet report...
I20260926 14:39:57.359742 24212 ts_tablet_manager.cc:1397] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 14:39:57.360159 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.359902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.195:47249)
0926 14:39:57.359999 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.360097 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 14:39:57.360232 24215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.358968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.194:52181)
0926 14:39:57.360111 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.360163 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1091,"thread_start_us":83,"threads_started":1}
I20260926 14:39:57.360579 24212 raft_consensus.cc:348] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } } peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } }
I20260926 14:39:57.360697 24212 raft_consensus.cc:374] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:57.360733 24212 raft_consensus.cc:729] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b35d833c2914abbb9412034f3541486, State: Initialized, Role: FOLLOWER
I20260926 14:39:57.360847 24212 consensus_queue.cc:260] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } } peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } }
I20260926 14:39:57.361161 24212 ts_tablet_manager.cc:1428] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:39:57.361763 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.361390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.193:37875)
0926 14:39:57.361611 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.361665 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260926 14:39:57.362407 23836 heartbeater.cc:503] Master 127.16.90.253:38709 was elected leader, sending a full tablet report...
I20260926 14:39:57.362579 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.362301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.193:44961)
0926 14:39:57.362479 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.362519 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 14:39:57.453317 24216 raft_consensus.cc:482] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:57.453450 24216 raft_consensus.cc:504] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } } peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } }
I20260926 14:39:57.453794 24216 leader_election.cc:288] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f22762d682134b62bbbcb5e765dbb000 (127.16.90.194:34647), 6b35d833c2914abbb9412034f3541486 (127.16.90.193:39189)
I20260926 14:39:57.454257 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da89bf5ba3504099b2260093cf0d528c" candidate_uuid: "603b1dae66d04c5baa64e503a0ae74d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f22762d682134b62bbbcb5e765dbb000" is_pre_election: true
I20260926 14:39:57.454285 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da89bf5ba3504099b2260093cf0d528c" candidate_uuid: "603b1dae66d04c5baa64e503a0ae74d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b35d833c2914abbb9412034f3541486" is_pre_election: true
I20260926 14:39:57.454425 23788 raft_consensus.cc:2454] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603b1dae66d04c5baa64e503a0ae74d1 in term 0.
I20260926 14:39:57.454445 23925 raft_consensus.cc:2454] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603b1dae66d04c5baa64e503a0ae74d1 in term 0.
I20260926 14:39:57.454751 23997 leader_election.cc:302] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603b1dae66d04c5baa64e503a0ae74d1, 6b35d833c2914abbb9412034f3541486; no voters:
I20260926 14:39:57.454944 24216 raft_consensus.cc:2781] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:57.455001 24216 raft_consensus.cc:482] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:57.455029 24216 raft_consensus.cc:3037] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:57.456071 24216 raft_consensus.cc:504] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } } peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } }
I20260926 14:39:57.456372 24216 leader_election.cc:288] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [CANDIDATE]: Term 1 election: Requested vote from peers f22762d682134b62bbbcb5e765dbb000 (127.16.90.194:34647), 6b35d833c2914abbb9412034f3541486 (127.16.90.193:39189)
I20260926 14:39:57.456687 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da89bf5ba3504099b2260093cf0d528c" candidate_uuid: "603b1dae66d04c5baa64e503a0ae74d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f22762d682134b62bbbcb5e765dbb000"
I20260926 14:39:57.456748 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da89bf5ba3504099b2260093cf0d528c" candidate_uuid: "603b1dae66d04c5baa64e503a0ae74d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b35d833c2914abbb9412034f3541486"
I20260926 14:39:57.456795 23925 raft_consensus.cc:3037] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:57.456867 23788 raft_consensus.cc:3037] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:57.457729 23925 raft_consensus.cc:2454] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603b1dae66d04c5baa64e503a0ae74d1 in term 1.
I20260926 14:39:57.457747 23788 raft_consensus.cc:2454] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603b1dae66d04c5baa64e503a0ae74d1 in term 1.
I20260926 14:39:57.457971 23997 leader_election.cc:302] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603b1dae66d04c5baa64e503a0ae74d1, 6b35d833c2914abbb9412034f3541486; no voters:
I20260926 14:39:57.458118 24216 raft_consensus.cc:2781] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:57.458339 24216 raft_consensus.cc:686] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 1 LEADER]: Becoming Leader. State: Replica: 603b1dae66d04c5baa64e503a0ae74d1, State: Running, Role: LEADER
I20260926 14:39:57.458585 24216 consensus_queue.cc:237] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } } peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } }
I20260926 14:39:57.458998 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.458778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.195:34023)
0926 14:39:57.458886 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.458939 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 14:39:57.461159 23469 catalog_manager.cc:5495] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 603b1dae66d04c5baa64e503a0ae74d1 (127.16.90.195). New cstate: current_term: 1 leader_uuid: "603b1dae66d04c5baa64e503a0ae74d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603b1dae66d04c5baa64e503a0ae74d1" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 } health_report { overall_health: UNKNOWN } } }
I20260926 14:39:57.500247 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.496893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.16.90.195:44307)
0926 14:39:57.497013 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:57.497017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:57.497031 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:39:57.497109 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:57.497111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:57.497163 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:57.497265 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:39:57.497486 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.497989 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.498874 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:57.499081 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:57.499165 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:57.499405 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:39:57.499878 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:39:57.499976 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:57.500042 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 14:39:57.500096 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:39:57.502830 23788 raft_consensus.cc:1261] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486 [term 1 FOLLOWER]: Refusing update from remote peer 603b1dae66d04c5baa64e503a0ae74d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:57.502996 23925 raft_consensus.cc:1261] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000 [term 1 FOLLOWER]: Refusing update from remote peer 603b1dae66d04c5baa64e503a0ae74d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:39:57.503311 24222 consensus_queue.cc:1035] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b35d833c2914abbb9412034f3541486" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:57.503453 24216 consensus_queue.cc:1035] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f22762d682134b62bbbcb5e765dbb000" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:39:57.505177 24219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.504872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.194:60135)
0926 14:39:57.505032 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.505096 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 14:39:57.506268 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.504987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.193:51069)
0926 14:39:57.506139 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.506197 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1084}
I20260926 14:39:57.508036 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.507804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.195:40819)
0926 14:39:57.507920 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.507972 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 14:39:57.509564 24219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.509315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.194:48521)
0926 14:39:57.509453 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.509502 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 14:39:57.509845 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.509584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.193:55367)
0926 14:39:57.509725 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.509777 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 14:39:57.510524 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.510169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44165 (local address 127.16.90.195:35591)
0926 14:39:57.510402 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.510457 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260926 14:39:57.517804 23470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:57.517954 23470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:39:57.521380 23470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16b721197e224140a9783e64705f54b6: Sending DeleteTablet for 3 replicas of tablet da89bf5ba3504099b2260093cf0d528c
I20260926 14:39:57.522204 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet da89bf5ba3504099b2260093cf0d528c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:57 UTC) from {username='oryx', principal='oryx/127.16.90.253@KRBTEST.COM'} at 127.0.0.1:50810
I20260926 14:39:57.522204 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet da89bf5ba3504099b2260093cf0d528c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:57 UTC) from {username='oryx', principal='oryx/127.16.90.253@KRBTEST.COM'} at 127.0.0.1:36486
I20260926 14:39:57.522338 24042 tablet_service.cc:1514] Processing DeleteTablet for tablet da89bf5ba3504099b2260093cf0d528c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:39:57 UTC) from {username='oryx', principal='oryx/127.16.90.253@KRBTEST.COM'} at 127.0.0.1:55972
I20260926 14:39:57.522984 24235 tablet_replica.cc:330] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486: stopping tablet replica
I20260926 14:39:57.523085 24235 raft_consensus.cc:2229] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:57.523195 24235 raft_consensus.cc:2258] T da89bf5ba3504099b2260093cf0d528c P 6b35d833c2914abbb9412034f3541486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:57.523838 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 23615
I20260926 14:39:57.524816 24236 tablet_replica.cc:330] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000: stopping tablet replica
I20260926 14:39:57.524938 24236 raft_consensus.cc:2229] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:39:57.525067 24236 raft_consensus.cc:2258] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:57.525658 24236 ts_tablet_manager.cc:1905] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:57.527536 24237 tablet_replica.cc:330] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1: stopping tablet replica
I20260926 14:39:57.527659 24237 raft_consensus.cc:2229] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:39:57.527768 24236 ts_tablet_manager.cc:1918] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:39:57.527838 24236 log.cc:1196] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/da89bf5ba3504099b2260093cf0d528c
I20260926 14:39:57.527946 24237 raft_consensus.cc:2258] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:39:57.528371 24236 ts_tablet_manager.cc:1939] T da89bf5ba3504099b2260093cf0d528c P f22762d682134b62bbbcb5e765dbb000: Deleting consensus metadata
I20260926 14:39:57.528529 24237 ts_tablet_manager.cc:1905] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:39:57.529047 23456 catalog_manager.cc:4841] TS f22762d682134b62bbbcb5e765dbb000 (127.16.90.194:34647): tablet da89bf5ba3504099b2260093cf0d528c (table test-table [id=31d81f1ab48b4580b062c11d6cba3ab5]) successfully deleted
I20260926 14:39:57.530808 24237 ts_tablet_manager.cc:1918] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:39:57.530886 24237 log.cc:1196] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/da89bf5ba3504099b2260093cf0d528c
I20260926 14:39:57.531118 24237 ts_tablet_manager.cc:1939] T da89bf5ba3504099b2260093cf0d528c P 603b1dae66d04c5baa64e503a0ae74d1: Deleting consensus metadata
I20260926 14:39:57.531760 23456 catalog_manager.cc:4841] TS 603b1dae66d04c5baa64e503a0ae74d1 (127.16.90.195:44307): tablet da89bf5ba3504099b2260093cf0d528c (table test-table [id=31d81f1ab48b4580b062c11d6cba3ab5]) successfully deleted
I20260926 14:39:57.532558 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 23842
I20260926 14:39:57.532810 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:57.532542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:39189 (local address 127.0.0.1:50814)
0926 14:39:57.532675 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:57.532730 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 14:39:57.533733 23456 catalog_manager.cc:4584] TS 6b35d833c2914abbb9412034f3541486 (127.16.90.193:39189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da89bf5ba3504099b2260093cf0d528c: Network error: Client connection negotiation failed: client connection to 127.16.90.193:39189: connect: Connection refused (error 111)
I20260926 14:39:57.538905 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 23979
I20260926 14:39:57.543754 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 23327
I20260926 14:39:57.548790 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 23434
2026-09-26T14:39:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5609 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:39:57 dist-test-slave-x4sn krb5kdc[24240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:39:57 dist-test-slave-x4sn krb5kdc[24240](info): set up 2 sockets
Sep 26 14:39:57 dist-test-slave-x4sn krb5kdc[24240](info): commencing operation
krb5kdc: starting...
W20260926 14:39:58.505587 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:39:58 dist-test-slave-x4sn krb5kdc[24240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:39:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.254.keytab.
Entry for principal oryx/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.254.keytab.
I20260926 14:39:58.618695 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35447
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.254.keytab
--principal=oryx/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:35447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:58.722957 24253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:58.723209 24253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:58.723246 24253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:58.727155 24253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:39:58.727206 24253 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 14:39:58.727221 24253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:58.727238 24253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:58.727252 24253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:39:58.727268 24253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:39:58.732326 24253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:35447
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.254.keytab
--principal=oryx/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:58.733299 24253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:58.734236 24253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:58.742291 24261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:58.742352 24259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:58.742300 24258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:58.742690 24253 server_base.cc:1034] running on GCE node
I20260926 14:39:58.743225 24253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:58.744328 24253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:58.745503 24253 hybrid_clock.cc:648] HybridClock initialized: now 1790433598745488 us; error 28 us; skew 500 ppm
Sep 26 14:39:58 dist-test-slave-x4sn krb5kdc[24240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433598, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:58.748347 24253 init.cc:377] Logged in from keytab as oryx/127.16.90.254@KRBTEST.COM (short username oryx)
I20260926 14:39:58.749264 24253 webserver.cc:460] Webserver started at http://127.16.90.254:34193/ using document root <none> and password file <none>
I20260926 14:39:58.749812 24253 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:58.749878 24253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:58.750074 24253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:58.752131 24253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "e063eace0a7149628b4caf51284e9d9a"
format_stamp: "Formatted at 2026-09-26 14:39:58 on dist-test-slave-x4sn"
server_key: "2ba8010ed5cca46013d2f3890fd6e5fe"
server_key_iv: "c58c001e6dc144d13ebdd72bf1509e94"
server_key_version: "encryptionkey@0"
I20260926 14:39:58.752667 24253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "e063eace0a7149628b4caf51284e9d9a"
format_stamp: "Formatted at 2026-09-26 14:39:58 on dist-test-slave-x4sn"
server_key: "2ba8010ed5cca46013d2f3890fd6e5fe"
server_key_iv: "c58c001e6dc144d13ebdd72bf1509e94"
server_key_version: "encryptionkey@0"
I20260926 14:39:58.756177 24253 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:39:58.758458 24268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:58.759617 24253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:58.759759 24253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "e063eace0a7149628b4caf51284e9d9a"
format_stamp: "Formatted at 2026-09-26 14:39:58 on dist-test-slave-x4sn"
server_key: "2ba8010ed5cca46013d2f3890fd6e5fe"
server_key_iv: "c58c001e6dc144d13ebdd72bf1509e94"
server_key_version: "encryptionkey@0"
I20260926 14:39:58.759850 24253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:58.776198 24253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:58.778826 24253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:58.778995 24253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:58.786593 24253 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:35447
I20260926 14:39:58.786600 24330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:35447 every 8 connection(s)
I20260926 14:39:58.787499 24253 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:39:58.790330 24332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:58.794004 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 24253
I20260926 14:39:58.794112 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:39:58.794375 16747 external_mini_cluster.cc:1428] Setting key 01822b24ffe68e4a39f8d9a325fccfd4
I20260926 14:39:58.796531 24332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a: Bootstrap starting.
I20260926 14:39:58.799145 24332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:58.799872 24332 log.cc:824] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:58.801821 24332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a: No bootstrap required, opened a new log
Sep 26 14:39:58 dist-test-slave-x4sn krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
I20260926 14:39:58.805624 24332 raft_consensus.cc:348] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063eace0a7149628b4caf51284e9d9a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35447 } }
I20260926 14:39:58.805840 24332 raft_consensus.cc:374] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:58.805874 24332 raft_consensus.cc:729] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063eace0a7149628b4caf51284e9d9a, State: Initialized, Role: FOLLOWER
I20260926 14:39:58.806316 24332 consensus_queue.cc:260] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063eace0a7149628b4caf51284e9d9a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35447 } }
I20260926 14:39:58.806442 24332 raft_consensus.cc:388] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:39:58.806505 24332 raft_consensus.cc:482] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:39:58.806605 24332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:58.807653 24332 raft_consensus.cc:504] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063eace0a7149628b4caf51284e9d9a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35447 } }
I20260926 14:39:58.808017 24332 leader_election.cc:302] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e063eace0a7149628b4caf51284e9d9a; no voters:
I20260926 14:39:58.808341 24332 leader_election.cc:288] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:39:58.808414 24337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:58.808645 24337 raft_consensus.cc:686] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [term 1 LEADER]: Becoming Leader. State: Replica: e063eace0a7149628b4caf51284e9d9a, State: Running, Role: LEADER
I20260926 14:39:58.808962 24337 consensus_queue.cc:237] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063eace0a7149628b4caf51284e9d9a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35447 } }
I20260926 14:39:58.809132 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:58.795591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.16.90.254:35447)
0926 14:39:58.796034 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:58.796052 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:58.796095 (+ 43us) server_negotiation.cc:407] Connection header received
0926 14:39:58.796891 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:58.796915 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:58.797308 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:58.797706 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:58.799025 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:58.800137 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:58.801223 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:58.801684 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:58.801815 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:58.804435 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:58.804440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:58.804452 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:58.804478 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:58.806147 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:58.806543 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:58.806550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:58.806555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:58.806710 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:58.806957 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:58.806960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:58.806961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:58.807447 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:58.807683 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:58.807842 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 14:39:58.808132 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260926 14:39:58.809547 24332 sys_catalog.cc:564] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:39:58.810600 24339 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e063eace0a7149628b4caf51284e9d9a. Latest consensus state: current_term: 1 leader_uuid: "e063eace0a7149628b4caf51284e9d9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063eace0a7149628b4caf51284e9d9a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35447 } } }
I20260926 14:39:58.810738 24339 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [sys.catalog]: This master's current role is: LEADER
I20260926 14:39:58.810600 24338 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e063eace0a7149628b4caf51284e9d9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063eace0a7149628b4caf51284e9d9a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35447 } } }
I20260926 14:39:58.810830 24338 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a [sys.catalog]: This master's current role is: LEADER
W20260926 14:39:58.813908 24352 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:39:58.813970 24352 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:39:58.814049 24346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:39:58.814738 24346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:39:58.820118 24346 catalog_manager.cc:1347] Generated new cluster ID: 00cbcbdb55e74b2d8b10c3950653a362
I20260926 14:39:58.820201 24346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:39:58.842865 24346 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:39:58.843839 24346 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:39:58.854297 24346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a: Generated new TSK 0
I20260926 14:39:58.855001 24346 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab.
Entry for principal oryx/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab.
I20260926 14:39:58.890542 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35447
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab
--principal=oryx/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:59.003208 24358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:59.003489 24358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:59.003540 24358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:59.007725 24358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:59.007787 24358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:59.007827 24358 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:39:59.007869 24358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:39:59.013238 24358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab
--principal=oryx/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.90.254:35447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:59.014328 24358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:59.015352 24358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:59.025060 24363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:59.025031 24364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:59.025149 24358 server_base.cc:1034] running on GCE node
W20260926 14:39:59.025031 24366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:59.025665 24358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:59.026269 24358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:59.027432 24358 hybrid_clock.cc:648] HybridClock initialized: now 1790433599027413 us; error 29 us; skew 500 ppm
Sep 26 14:39:59 dist-test-slave-x4sn krb5kdc[24240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433599, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:59.030061 24358 init.cc:377] Logged in from keytab as oryx/127.16.90.193@KRBTEST.COM (short username oryx)
I20260926 14:39:59.030917 24358 webserver.cc:460] Webserver started at http://127.16.90.193:36415/ using document root <none> and password file <none>
I20260926 14:39:59.031430 24358 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:59.031493 24358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:59.031733 24358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:59.033690 24358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "2e21628c67d745619e80038e9cb14fe8"
format_stamp: "Formatted at 2026-09-26 14:39:59 on dist-test-slave-x4sn"
server_key: "7f5c366e1ec73680a39924ed5a424967"
server_key_iv: "2079e40bc32d02e19b10de4838950e3e"
server_key_version: "encryptionkey@0"
I20260926 14:39:59.034207 24358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "2e21628c67d745619e80038e9cb14fe8"
format_stamp: "Formatted at 2026-09-26 14:39:59 on dist-test-slave-x4sn"
server_key: "7f5c366e1ec73680a39924ed5a424967"
server_key_iv: "2079e40bc32d02e19b10de4838950e3e"
server_key_version: "encryptionkey@0"
I20260926 14:39:59.037738 24358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 14:39:59.040087 24373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:59.041311 24358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:59.041424 24358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "2e21628c67d745619e80038e9cb14fe8"
format_stamp: "Formatted at 2026-09-26 14:39:59 on dist-test-slave-x4sn"
server_key: "7f5c366e1ec73680a39924ed5a424967"
server_key_iv: "2079e40bc32d02e19b10de4838950e3e"
server_key_version: "encryptionkey@0"
I20260926 14:39:59.041519 24358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:59.070521 24358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:59.073524 24358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:59.073712 24358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:59.075173 24358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:59.075223 24358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:59.075276 24358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:59.075305 24358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:39:59 dist-test-slave-x4sn krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433599, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
I20260926 14:39:59.085268 24358 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:45331
I20260926 14:39:59.085294 24488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:45331 every 8 connection(s)
I20260926 14:39:59.086117 24358 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:39:59.086992 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 24358
I20260926 14:39:59.087116 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:39:59.087394 16747 external_mini_cluster.cc:1428] Setting key 55761c4434ed1caa89b30ec77068634d
I20260926 14:39:59.088400 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.076133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:35683 (local address 127.16.90.254:35447)
0926 14:39:59.076249 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.076253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.077303 (+ 1050us) server_negotiation.cc:407] Connection header received
0926 14:39:59.078373 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.078378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.078439 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.078540 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:59.079995 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.080816 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.082066 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.082296 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.082388 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.084794 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:59.084798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:59.084801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:59.084830 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:59.086293 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.086853 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.086857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.086859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.086948 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.087298 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.087301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.087303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.087498 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:59.087601 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:59.088169 (+ 568us) server_negotiation.cc:299] Negotiation successful
0926 14:39:59.088277 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:39:59.089282 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.076673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35447 (local address 127.16.90.193:35683)
0926 14:39:59.077155 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:59.077199 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:59.078132 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:59.078734 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:59.078743 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:59.079164 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:59.079837 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.079858 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.080943 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.080947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.081952 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.081961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.082447 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.082451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.082622 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.083140 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:59.083155 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:59.084638 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:59.086393 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:59.086398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:59.086412 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:59.086738 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:59.087032 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:59.087035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:59.087037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:59.087209 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:59.087601 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:59.087607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:59.087905 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 14:39:59.088101 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 14:39:59.088404 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":135,"threads_started":1}
I20260926 14:39:59.090708 24489 heartbeater.cc:347] Connected to a master server at 127.16.90.254:35447
I20260926 14:39:59.090847 24489 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:59.091380 24489 heartbeater.cc:511] Master 127.16.90.254:35447 requested a full tablet report, sending...
I20260926 14:39:59.092957 24284 ts_manager.cc:194] Registered new tserver with Master: 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331)
I20260926 14:39:59.094278 24284 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:35683
WARNING: no policy specified for oryx/127.16.90.194@KRBTEST.COM; defaulting to no policy
I20260926 14:39:59.099599 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.092829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:55919 (local address 127.16.90.254:35447)
0926 14:39:59.092932 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.092936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.093149 (+ 213us) server_negotiation.cc:407] Connection header received
0926 14:39:59.093261 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.093264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.093309 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.093384 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:59.093867 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.094636 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.095480 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.095725 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.095811 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.096802 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:59.096805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:59.096808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:59.096831 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:59.098166 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.098537 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.098540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.098542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.098614 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.098854 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.098858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.098859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.099031 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:59.099119 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:59.099349 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 14:39:59.099452 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":36}
I20260926 14:39:59.099607 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.092777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35447 (local address 127.16.90.193:55919)
0926 14:39:59.093079 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:59.093093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:59.093184 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:59.093410 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:59.093413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:59.093609 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:59.093757 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.093764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.094777 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.094780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.095373 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.095382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.095816 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.095820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.095920 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.096356 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:59.096369 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:59.096669 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:59.098268 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:59.098271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:59.098273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:59.098454 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:59.098698 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:59.098701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:59.098703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:59.098777 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:59.099120 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:59.099123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:59.099208 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 14:39:59.099289 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 14:39:59.099416 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab.
Entry for principal oryx/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab.
I20260926 14:39:59.115211 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35447
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab
--principal=oryx/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:59.221196 24498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:59.221451 24498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:59.221514 24498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:59.225551 24498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:59.225617 24498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:59.225672 24498 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:39:59.225732 24498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:39:59.230939 24498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab
--principal=oryx/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.90.254:35447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:59.232023 24498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:59.233018 24498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:59.242281 24506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:59.242306 24503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:59.242364 24498 server_base.cc:1034] running on GCE node
W20260926 14:39:59.242298 24504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:59.242861 24498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:59.243444 24498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:59.244591 24498 hybrid_clock.cc:648] HybridClock initialized: now 1790433599244565 us; error 39 us; skew 500 ppm
Sep 26 14:39:59 dist-test-slave-x4sn krb5kdc[24240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433599, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:59.247140 24498 init.cc:377] Logged in from keytab as oryx/127.16.90.194@KRBTEST.COM (short username oryx)
I20260926 14:39:59.248063 24498 webserver.cc:460] Webserver started at http://127.16.90.194:36375/ using document root <none> and password file <none>
I20260926 14:39:59.248572 24498 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:59.248636 24498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:59.248835 24498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:59.250828 24498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "ede771f9fef84a1a81651cf04d3dece2"
format_stamp: "Formatted at 2026-09-26 14:39:59 on dist-test-slave-x4sn"
server_key: "db5cdb9c34948076d7e9eeb7008df07e"
server_key_iv: "a865376cdcb0cca8367363bfd1d7be91"
server_key_version: "encryptionkey@0"
I20260926 14:39:59.251330 24498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "ede771f9fef84a1a81651cf04d3dece2"
format_stamp: "Formatted at 2026-09-26 14:39:59 on dist-test-slave-x4sn"
server_key: "db5cdb9c34948076d7e9eeb7008df07e"
server_key_iv: "a865376cdcb0cca8367363bfd1d7be91"
server_key_version: "encryptionkey@0"
I20260926 14:39:59.254884 24498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:39:59.257306 24513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:59.258592 24498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:39:59.258716 24498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "ede771f9fef84a1a81651cf04d3dece2"
format_stamp: "Formatted at 2026-09-26 14:39:59 on dist-test-slave-x4sn"
server_key: "db5cdb9c34948076d7e9eeb7008df07e"
server_key_iv: "a865376cdcb0cca8367363bfd1d7be91"
server_key_version: "encryptionkey@0"
I20260926 14:39:59.258811 24498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:59.275163 24498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:59.278321 24498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:59.278498 24498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:59.279953 24498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:59.280001 24498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:59.280055 24498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:59.280088 24498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:39:59 dist-test-slave-x4sn krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433599, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
I20260926 14:39:59.290222 24498 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:41703
I20260926 14:39:59.290292 24630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:41703 every 8 connection(s)
I20260926 14:39:59.291260 24498 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:39:59.293973 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.280954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:42641 (local address 127.16.90.254:35447)
0926 14:39:59.281130 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.281135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.282226 (+ 1091us) server_negotiation.cc:407] Connection header received
0926 14:39:59.283369 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.283373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.283436 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.283528 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:59.284866 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.285483 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.286643 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.286886 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.286989 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.290456 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:59.290460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:59.290463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:59.290492 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:59.292095 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.292653 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.292657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.292658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.292759 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.293049 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.293051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.293052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.293233 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:59.293304 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:59.293754 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 14:39:59.293855 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 14:39:59.294739 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.281496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35447 (local address 127.16.90.194:42641)
0926 14:39:59.282072 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:59.282116 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:59.283077 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:59.283776 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:59.283783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:59.284157 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:59.284707 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.284721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.285608 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.285613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.286514 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.286526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.287055 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.287059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.287215 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.288455 (+ 1240us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:59.288476 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:59.290273 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:59.292200 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:59.292206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:59.292220 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:59.292537 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:59.292840 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:59.292843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:59.292844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:59.292977 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:59.293309 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:59.293314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:59.293552 (+ 238us) client_negotiation.cc:769] Sending connection context
0926 14:39:59.293712 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 14:39:59.293955 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":149,"threads_started":1}
I20260926 14:39:59.295926 24631 heartbeater.cc:347] Connected to a master server at 127.16.90.254:35447
I20260926 14:39:59.296074 24631 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:59.296639 24631 heartbeater.cc:511] Master 127.16.90.254:35447 requested a full tablet report, sending...
I20260926 14:39:59.297786 24285 ts_manager.cc:194] Registered new tserver with Master: ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194:41703)
I20260926 14:39:59.298588 24285 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:42641
I20260926 14:39:59.300961 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 24498
I20260926 14:39:59.301074 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:39:59.301379 16747 external_mini_cluster.cc:1428] Setting key f176f1b61ebeaa5cfdc3c49d2aa7da54
I20260926 14:39:59.305111 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.298114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35447 (local address 127.16.90.194:47781)
0926 14:39:59.298439 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:59.298456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:59.298577 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:59.298820 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:59.298823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:59.299025 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:59.299139 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.299146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.300162 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.300164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.300742 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.300752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.301256 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.301260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.301349 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.301808 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:59.301819 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:59.302100 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:59.303647 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:59.303650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:59.303653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:59.303868 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:59.304139 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:59.304143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:59.304145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:59.304257 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:59.304695 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:59.304697 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:59.304773 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 14:39:59.304858 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:39:59.304970 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20260926 14:39:59.305164 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.298186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:47781 (local address 127.16.90.254:35447)
0926 14:39:59.298292 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.298297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.298517 (+ 220us) server_negotiation.cc:407] Connection header received
0926 14:39:59.298665 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.298669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.298721 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.298801 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:59.299281 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.300052 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.300849 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.301142 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.301238 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.302212 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:59.302214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:59.302216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:59.302239 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:59.303550 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.303954 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.303957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.303959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.304033 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.304347 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.304350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.304351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.304609 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:59.304711 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:59.304914 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 14:39:59.305019 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for oryx/127.16.90.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab.
Entry for principal oryx/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab.
I20260926 14:39:59.325995 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35447
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab
--principal=oryx/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:39:59.431159 24640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:39:59.431437 24640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:39:59.431483 24640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:39:59.435372 24640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:39:59.435426 24640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:39:59.435462 24640 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:39:59.435498 24640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:39:59.440506 24640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab
--principal=oryx/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.90.254:35447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:39:59.441437 24640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:39:59.442351 24640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:39:59.451171 24645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:59.451237 24646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:39:59.451190 24648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:39:59.451237 24640 server_base.cc:1034] running on GCE node
I20260926 14:39:59.451828 24640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:39:59.452500 24640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:39:59.453670 24640 hybrid_clock.cc:648] HybridClock initialized: now 1790433599453650 us; error 28 us; skew 500 ppm
Sep 26 14:39:59 dist-test-slave-x4sn krb5kdc[24240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433599, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:39:59.456256 24640 init.cc:377] Logged in from keytab as oryx/127.16.90.195@KRBTEST.COM (short username oryx)
I20260926 14:39:59.457031 24640 webserver.cc:460] Webserver started at http://127.16.90.195:39817/ using document root <none> and password file <none>
I20260926 14:39:59.457540 24640 fs_manager.cc:359] Metadata directory not provided
I20260926 14:39:59.457602 24640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:39:59.457783 24640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:39:59.459738 24640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "273d2eea96f2470fb3abe9f63766735c"
format_stamp: "Formatted at 2026-09-26 14:39:59 on dist-test-slave-x4sn"
server_key: "386d7a404e34a1d5ed2944735ccbfb66"
server_key_iv: "31bc7c7ff6d2a56f48943cc14b9fff4e"
server_key_version: "encryptionkey@0"
I20260926 14:39:59.460255 24640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "273d2eea96f2470fb3abe9f63766735c"
format_stamp: "Formatted at 2026-09-26 14:39:59 on dist-test-slave-x4sn"
server_key: "386d7a404e34a1d5ed2944735ccbfb66"
server_key_iv: "31bc7c7ff6d2a56f48943cc14b9fff4e"
server_key_version: "encryptionkey@0"
I20260926 14:39:59.464017 24640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260926 14:39:59.466464 24655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:59.467823 24640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:39:59.467914 24640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "273d2eea96f2470fb3abe9f63766735c"
format_stamp: "Formatted at 2026-09-26 14:39:59 on dist-test-slave-x4sn"
server_key: "386d7a404e34a1d5ed2944735ccbfb66"
server_key_iv: "31bc7c7ff6d2a56f48943cc14b9fff4e"
server_key_version: "encryptionkey@0"
I20260926 14:39:59.467993 24640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:39:59.476893 24640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:39:59.479457 24640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:39:59.479594 24640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:39:59.480892 24640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:39:59.480937 24640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:59.480976 24640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:39:59.481014 24640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:39:59.490242 24640 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:41457
Sep 26 14:39:59 dist-test-slave-x4sn krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433599, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
I20260926 14:39:59.490587 24770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:41457 every 8 connection(s)
I20260926 14:39:59.491185 24640 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:39:59.491387 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 24640
I20260926 14:39:59.491509 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:39:59.491871 16747 external_mini_cluster.cc:1428] Setting key 1247506a641e8bffc7036e5976e1d14c
I20260926 14:39:59.494642 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.481694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:54613 (local address 127.16.90.254:35447)
0926 14:39:59.481818 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.481821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.482818 (+ 997us) server_negotiation.cc:407] Connection header received
0926 14:39:59.483858 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.483862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.483923 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.483982 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:59.485571 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.486379 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.487969 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.488198 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.488290 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.491034 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:59.491038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:59.491040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:59.491068 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:59.492723 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.493227 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.493230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.493232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.493320 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.493680 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.493683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.493684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.493857 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:59.493931 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:59.494430 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 14:39:59.494529 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:39:59.495354 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.482206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35447 (local address 127.16.90.195:54613)
0926 14:39:59.482670 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:59.482709 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:59.483581 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:59.484408 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:59.484417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:59.484802 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:59.485430 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.485449 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.486504 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.486507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.487854 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.487863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.488334 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.488338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.488524 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.489138 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:59.489158 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:59.490873 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:59.492832 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:59.492838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:59.492852 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:59.493111 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:59.493406 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:59.493409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:59.493411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:59.493582 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:59.493941 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:59.493948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:59.494194 (+ 246us) client_negotiation.cc:769] Sending connection context
0926 14:39:59.494348 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 14:39:59.494590 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":129,"threads_started":1}
I20260926 14:39:59.496382 24771 heartbeater.cc:347] Connected to a master server at 127.16.90.254:35447
I20260926 14:39:59.496515 24771 heartbeater.cc:464] Registering TS with master...
I20260926 14:39:59.497035 24771 heartbeater.cc:511] Master 127.16.90.254:35447 requested a full tablet report, sending...
I20260926 14:39:59.498143 24284 ts_manager.cc:194] Registered new tserver with Master: 273d2eea96f2470fb3abe9f63766735c (127.16.90.195:41457)
I20260926 14:39:59.498883 24284 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:54613
I20260926 14:39:59.505048 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.498572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:53399 (local address 127.16.90.254:35447)
0926 14:39:59.498681 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.498686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.498831 (+ 145us) server_negotiation.cc:407] Connection header received
0926 14:39:59.498996 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.498999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.499045 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.499117 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:59.499589 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.500375 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.501173 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.501381 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.501456 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.502350 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:59.502352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:59.502354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:59.502373 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:59.503576 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.503963 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.503966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.503967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.504037 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.504312 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.504316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.504317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.504512 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:59.504599 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:59.504842 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 14:39:59.504931 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:39:59.505077 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.498469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35447 (local address 127.16.90.195:53399)
0926 14:39:59.498755 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:59.498769 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:59.498893 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:59.499145 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:59.499148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:59.499375 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:39:59.499488 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.499494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.500501 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.500504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.501076 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.501083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.501463 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.501466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.501528 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.501954 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:39:59.501965 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:39:59.502241 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:39:59.503705 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:59.503709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:59.503711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:59.503883 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:59.504117 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:39:59.504119 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:39:59.504121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:39:59.504212 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:39:59.504601 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:39:59.504603 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:39:59.504684 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 14:39:59.504782 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 14:39:59.504914 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260926 14:39:59.506454 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:39:59 dist-test-slave-x4sn krb5kdc[24240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:39:59 dist-test-slave-x4sn krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:39:59 dist-test-slave-x4sn krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:39:59.527565 24335 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:39:59.522191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.16.90.254:35447)
0926 14:39:59.522306 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.522309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.522320 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:39:59.522356 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.522358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.522393 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.522459 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:59.523059 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.523567 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.524375 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.524585 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.524662 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.527447 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41912: BlockingRecv error: recv got EOF from 127.0.0.1:41912 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 14:39:59 dist-test-slave-x4sn krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
I20260926 14:39:59.536003 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.528867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.16.90.254:35447)
0926 14:39:59.528983 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.528987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.529055 (+ 68us) server_negotiation.cc:407] Connection header received
0926 14:39:59.529181 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.529184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.529224 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.529281 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:39:59.529823 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.530316 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.531069 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.531353 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.531456 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.533452 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:39:59.533455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:39:59.533458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:39:59.533482 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:39:59.534557 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.534923 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.534926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.534928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.535007 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:39:59.535236 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:39:59.535239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:39:59.535241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:39:59.535405 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:39:59.535517 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:59.535747 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 14:39:59.535854 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:39:59.538441 24284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:39:59.541419 24284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:39:59.554836 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.548906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:41703 (local address 127.0.0.1:38192)
0926 14:39:59.549771 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:59.549818 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:59.549929 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:59.550842 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:59.550844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:59.550859 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:59.550929 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.550935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.552288 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.552291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.553337 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.553347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.554319 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.554322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.554421 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.554456 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 14:39:59.554559 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 14:39:59.554690 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260926 14:39:59.555047 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.549854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.16.90.193:45331)
0926 14:39:59.550189 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.550226 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.550246 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:39:59.550361 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.550365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.550527 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.550683 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:59.550946 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.552128 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.553736 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.554250 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.554348 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.554499 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:59.554811 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 14:39:59.554908 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":61,"threads_started":1}
I20260926 14:39:59.555126 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.549209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.16.90.195:41457)
0926 14:39:59.549584 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.549596 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.550228 (+ 632us) server_negotiation.cc:407] Connection header received
0926 14:39:59.550317 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.550323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.550527 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.550683 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:59.551136 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.551897 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.553696 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.554442 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.554535 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.554692 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:59.554917 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 14:39:59.555002 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":93,"threads_started":1}
I20260926 14:39:59.555380 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.549037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.16.90.194:41703)
0926 14:39:59.550336 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.550349 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.550368 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:39:59.550429 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.550432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.550577 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.550714 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:59.551040 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.552202 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.553454 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.554197 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.554305 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.554463 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:59.555138 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 14:39:59.555245 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":132,"threads_started":1}
I20260926 14:39:59.555413 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.549668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:45331 (local address 127.0.0.1:40578)
0926 14:39:59.550172 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:59.550185 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:59.550257 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:59.550689 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:59.550694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:59.550716 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:59.550838 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.550844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.552242 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.552245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.553642 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.553650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.554559 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.554562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.554645 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.554694 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 14:39:59.554762 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:39:59.554819 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":59,"thread_start_us":90,"threads_started":1}
I20260926 14:39:59.555701 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.548988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:41457 (local address 127.0.0.1:34004)
0926 14:39:59.550011 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:59.550024 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:59.550114 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:59.550945 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:59.550948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:59.550961 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:59.551032 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.551038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.551995 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.551998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.553564 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.553574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.554536 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.554540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.554637 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.554701 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 14:39:59.554770 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:39:59.554844 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":136,"thread_start_us":78,"threads_started":1}
I20260926 14:39:59.558243 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 900f75fd056a40b99f3ffac20e8ad158 (DEFAULT_TABLE table=test-table [id=6e74f6724e264b01b6d70836204a6998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:59.558388 24565 tablet_service.cc:1467] Processing CreateTablet for tablet 900f75fd056a40b99f3ffac20e8ad158 (DEFAULT_TABLE table=test-table [id=6e74f6724e264b01b6d70836204a6998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:59.558483 24423 tablet_service.cc:1467] Processing CreateTablet for tablet 900f75fd056a40b99f3ffac20e8ad158 (DEFAULT_TABLE table=test-table [id=6e74f6724e264b01b6d70836204a6998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:39:59.559185 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900f75fd056a40b99f3ffac20e8ad158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:59.559374 24565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900f75fd056a40b99f3ffac20e8ad158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:59.559374 24423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900f75fd056a40b99f3ffac20e8ad158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:39:59.565696 24799 tablet_bootstrap.cc:492] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c: Bootstrap starting.
I20260926 14:39:59.566993 24800 tablet_bootstrap.cc:492] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8: Bootstrap starting.
I20260926 14:39:59.567934 24801 tablet_bootstrap.cc:492] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2: Bootstrap starting.
I20260926 14:39:59.568377 24799 tablet_bootstrap.cc:654] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:59.569025 24800 tablet_bootstrap.cc:654] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:59.569216 24799 log.cc:824] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:59.569559 24801 tablet_bootstrap.cc:654] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2: Neither blocks nor log segments found. Creating new log.
I20260926 14:39:59.569828 24800 log.cc:824] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:59.570390 24801 log.cc:824] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2: Log is configured to *not* fsync() on all Append() calls
I20260926 14:39:59.571301 24799 tablet_bootstrap.cc:492] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c: No bootstrap required, opened a new log
I20260926 14:39:59.571451 24799 ts_tablet_manager.cc:1397] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:39:59.572033 24801 tablet_bootstrap.cc:492] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2: No bootstrap required, opened a new log
I20260926 14:39:59.572206 24801 ts_tablet_manager.cc:1397] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260926 14:39:59.572978 24800 tablet_bootstrap.cc:492] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8: No bootstrap required, opened a new log
I20260926 14:39:59.573154 24800 ts_tablet_manager.cc:1397] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 14:39:59.574927 24801 raft_consensus.cc:348] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } }
I20260926 14:39:59.575139 24801 raft_consensus.cc:374] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:59.575191 24801 raft_consensus.cc:729] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede771f9fef84a1a81651cf04d3dece2, State: Initialized, Role: FOLLOWER
I20260926 14:39:59.575186 24799 raft_consensus.cc:348] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } }
I20260926 14:39:59.575430 24799 raft_consensus.cc:374] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:59.575486 24799 raft_consensus.cc:729] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273d2eea96f2470fb3abe9f63766735c, State: Initialized, Role: FOLLOWER
I20260926 14:39:59.575599 24801 consensus_queue.cc:260] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } }
I20260926 14:39:59.575906 24799 consensus_queue.cc:260] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } }
I20260926 14:39:59.576301 24631 heartbeater.cc:503] Master 127.16.90.254:35447 was elected leader, sending a full tablet report...
I20260926 14:39:59.576530 24801 ts_tablet_manager.cc:1428] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 14:39:59.576913 24800 raft_consensus.cc:348] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } }
I20260926 14:39:59.577147 24800 raft_consensus.cc:374] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:39:59.577204 24800 raft_consensus.cc:729] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e21628c67d745619e80038e9cb14fe8, State: Initialized, Role: FOLLOWER
I20260926 14:39:59.577561 24771 heartbeater.cc:503] Master 127.16.90.254:35447 was elected leader, sending a full tablet report...
I20260926 14:39:59.577570 24800 consensus_queue.cc:260] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } }
I20260926 14:39:59.578094 24799 ts_tablet_manager.cc:1428] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 14:39:59.578402 24800 ts_tablet_manager.cc:1428] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 14:39:59.578574 24489 heartbeater.cc:503] Master 127.16.90.254:35447 was elected leader, sending a full tablet report...
W20260926 14:39:59.587262 24490 tablet.cc:2369] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:39:59.742532 24772 tablet.cc:2369] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:39:59.792418 24632 tablet.cc:2369] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:39:59.932987 24806 raft_consensus.cc:482] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:39:59.933224 24806 raft_consensus.cc:504] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } }
I20260926 14:39:59.934341 24806 leader_election.cc:288] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194:41703), 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331)
I20260926 14:39:59.938618 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.934607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:41703 (local address 127.16.90.195:33965)
0926 14:39:59.934734 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:59.934749 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:59.934836 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:59.935077 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:59.935080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:59.935101 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:59.935221 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.935226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.936228 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.936232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.937360 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.937368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.938195 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.938198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.938304 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.938329 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:39:59.938404 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 14:39:59.938475 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":1}
I20260926 14:39:59.938632 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.934681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:33965 (local address 127.16.90.194:41703)
0926 14:39:59.934804 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.934808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.934825 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:39:59.934927 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.934929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.934972 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.935036 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:59.935352 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.936123 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.937482 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.938084 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.938209 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.938264 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:59.938460 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 14:39:59.938531 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:39:59.939227 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900f75fd056a40b99f3ffac20e8ad158" candidate_uuid: "273d2eea96f2470fb3abe9f63766735c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede771f9fef84a1a81651cf04d3dece2" is_pre_election: true
I20260926 14:39:59.939345 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.934698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:51785 (local address 127.16.90.193:45331)
0926 14:39:59.934818 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:39:59.934823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:39:59.935364 (+ 541us) server_negotiation.cc:407] Connection header received
0926 14:39:59.935465 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:39:59.935467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:39:59.935507 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:39:59.935562 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:39:59.936266 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.937026 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.938315 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.938867 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.938972 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.939018 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:39:59.939174 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 14:39:59.939242 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:39:59.939366 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:39:59.934608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:45331 (local address 127.16.90.195:51785)
0926 14:39:59.935274 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0926 14:39:59.935287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:39:59.935384 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:39:59.935585 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:39:59.935588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:39:59.935604 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:39:59.936151 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.936160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.937125 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.937128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.938202 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:39:59.938211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:39:59.938982 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:39:59.938986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:39:59.939073 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:39:59.939089 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 14:39:59.939159 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 14:39:59.939222 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":305,"thread_start_us":109,"threads_started":1}
I20260926 14:39:59.939549 24585 raft_consensus.cc:2454] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 273d2eea96f2470fb3abe9f63766735c in term 0.
I20260926 14:39:59.939960 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900f75fd056a40b99f3ffac20e8ad158" candidate_uuid: "273d2eea96f2470fb3abe9f63766735c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e21628c67d745619e80038e9cb14fe8" is_pre_election: true
I20260926 14:39:59.940084 24656 leader_election.cc:302] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 273d2eea96f2470fb3abe9f63766735c, ede771f9fef84a1a81651cf04d3dece2; no voters:
I20260926 14:39:59.940244 24443 raft_consensus.cc:2454] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 273d2eea96f2470fb3abe9f63766735c in term 0.
I20260926 14:39:59.940325 24806 raft_consensus.cc:2781] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:39:59.940400 24806 raft_consensus.cc:482] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:39:59.940450 24806 raft_consensus.cc:3037] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:59.941462 24806 raft_consensus.cc:504] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } }
I20260926 14:39:59.941813 24806 leader_election.cc:288] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [CANDIDATE]: Term 1 election: Requested vote from peers ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194:41703), 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331)
I20260926 14:39:59.942090 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900f75fd056a40b99f3ffac20e8ad158" candidate_uuid: "273d2eea96f2470fb3abe9f63766735c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede771f9fef84a1a81651cf04d3dece2"
I20260926 14:39:59.942147 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900f75fd056a40b99f3ffac20e8ad158" candidate_uuid: "273d2eea96f2470fb3abe9f63766735c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e21628c67d745619e80038e9cb14fe8"
I20260926 14:39:59.942214 24585 raft_consensus.cc:3037] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:59.942246 24443 raft_consensus.cc:3037] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:39:59.943378 24585 raft_consensus.cc:2454] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 273d2eea96f2470fb3abe9f63766735c in term 1.
I20260926 14:39:59.943382 24443 raft_consensus.cc:2454] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 273d2eea96f2470fb3abe9f63766735c in term 1.
I20260926 14:39:59.943689 24658 leader_election.cc:302] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 273d2eea96f2470fb3abe9f63766735c, 2e21628c67d745619e80038e9cb14fe8; no voters:
I20260926 14:39:59.943852 24806 raft_consensus.cc:2781] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:39:59.944047 24806 raft_consensus.cc:686] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 1 LEADER]: Becoming Leader. State: Replica: 273d2eea96f2470fb3abe9f63766735c, State: Running, Role: LEADER
I20260926 14:39:59.944347 24806 consensus_queue.cc:237] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } }
I20260926 14:39:59.947400 24284 catalog_manager.cc:5495] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c reported cstate change: term changed from 0 to 1, leader changed from <none> to 273d2eea96f2470fb3abe9f63766735c (127.16.90.195). New cstate: current_term: 1 leader_uuid: "273d2eea96f2470fb3abe9f63766735c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } health_report { overall_health: UNKNOWN } } }
I20260926 14:40:00.004042 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.000667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.16.90.195:41457)
0926 14:40:00.000806 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.000810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.000825 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:40:00.000899 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.000901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.000953 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.001047 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:40:00.001298 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.001815 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.002645 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.002861 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.002946 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.003205 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.003643 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.003768 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.003844 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.003907 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:40:00.009955 24585 raft_consensus.cc:1261] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Refusing update from remote peer 273d2eea96f2470fb3abe9f63766735c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:40:00.009955 24443 raft_consensus.cc:1261] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Refusing update from remote peer 273d2eea96f2470fb3abe9f63766735c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:40:00.010687 24806 consensus_queue.cc:1035] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:40:00.010914 24809 consensus_queue.cc:1035] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:40:00.017859 24812 mvcc.cc:205] Tried to move back new op lower bound from 7333616025633079296 to 7333616025374519296. Current Snapshot: MvccSnapshot[applied={T|T < 7333616025633079296}]
I20260926 14:40:00.021021 24813 mvcc.cc:205] Tried to move back new op lower bound from 7333616025633079296 to 7333616025374519296. Current Snapshot: MvccSnapshot[applied={T|T < 7333616025633079296 or (T in {7333616025633079296})}]
I20260926 14:40:00.022063 24815 mvcc.cc:205] Tried to move back new op lower bound from 7333616025633079296 to 7333616025374519296. Current Snapshot: MvccSnapshot[applied={T|T < 7333616025633079296}]
I20260926 14:40:00.025321 24285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:40:00.025465 24285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:40:00.027634 24285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a: Sending DeleteTablet for 3 replicas of tablet 900f75fd056a40b99f3ffac20e8ad158
I20260926 14:40:00.028335 24565 tablet_service.cc:1514] Processing DeleteTablet for tablet 900f75fd056a40b99f3ffac20e8ad158 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:00 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:38192
I20260926 14:40:00.028504 24423 tablet_service.cc:1514] Processing DeleteTablet for tablet 900f75fd056a40b99f3ffac20e8ad158 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:00 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:40578
I20260926 14:40:00.028657 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 900f75fd056a40b99f3ffac20e8ad158 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:00 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:34004
I20260926 14:40:00.028935 24821 tablet_replica.cc:330] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2: stopping tablet replica
I20260926 14:40:00.029279 24822 tablet_replica.cc:330] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8: stopping tablet replica
I20260926 14:40:00.029381 24821 raft_consensus.cc:2229] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:40:00.029511 24822 raft_consensus.cc:2229] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:40:00.029544 24285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:40:00.029654 24821 raft_consensus.cc:2258] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:40:00.029723 24822 raft_consensus.cc:2258] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:40:00.029922 24285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:40:00.031694 24821 ts_tablet_manager.cc:1905] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:00.031904 24822 ts_tablet_manager.cc:1905] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:00.034588 24822 ts_tablet_manager.cc:1918] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:40:00.034745 24822 log.cc:1196] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/wal/wals/900f75fd056a40b99f3ffac20e8ad158
I20260926 14:40:00.034750 24821 ts_tablet_manager.cc:1918] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:40:00.034864 24821 log.cc:1196] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/900f75fd056a40b99f3ffac20e8ad158
I20260926 14:40:00.035184 24821 ts_tablet_manager.cc:1939] T 900f75fd056a40b99f3ffac20e8ad158 P ede771f9fef84a1a81651cf04d3dece2: Deleting consensus metadata
I20260926 14:40:00.035328 24822 ts_tablet_manager.cc:1939] T 900f75fd056a40b99f3ffac20e8ad158 P 2e21628c67d745619e80038e9cb14fe8: Deleting consensus metadata
I20260926 14:40:00.035998 24423 tablet_service.cc:1467] Processing CreateTablet for tablet d71664f0db5047f794234182ee32d1a1 (DEFAULT_TABLE table=test-table [id=8ea66853c9674ad6832ce41c979c5bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:40:00.036273 24423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71664f0db5047f794234182ee32d1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:00.036569 24271 catalog_manager.cc:4841] TS 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331): tablet 900f75fd056a40b99f3ffac20e8ad158 (table test-table [id=6e74f6724e264b01b6d70836204a6998]) successfully deleted
I20260926 14:40:00.037798 24800 tablet_bootstrap.cc:492] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8: Bootstrap starting.
I20260926 14:40:00.037963 24565 tablet_service.cc:1467] Processing CreateTablet for tablet d71664f0db5047f794234182ee32d1a1 (DEFAULT_TABLE table=test-table [id=8ea66853c9674ad6832ce41c979c5bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:40:00.038203 24565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71664f0db5047f794234182ee32d1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:00.038681 24800 tablet_bootstrap.cc:654] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:00.040155 24801 tablet_bootstrap.cc:492] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2: Bootstrap starting.
I20260926 14:40:00.040467 24269 catalog_manager.cc:4841] TS ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194:41703): tablet 900f75fd056a40b99f3ffac20e8ad158 (table test-table [id=6e74f6724e264b01b6d70836204a6998]) successfully deleted
I20260926 14:40:00.041015 24801 tablet_bootstrap.cc:654] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:00.041294 24705 tablet_service.cc:1467] Processing CreateTablet for tablet d71664f0db5047f794234182ee32d1a1 (DEFAULT_TABLE table=test-table [id=8ea66853c9674ad6832ce41c979c5bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:40:00.041615 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71664f0db5047f794234182ee32d1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:00.042322 24800 tablet_bootstrap.cc:492] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8: No bootstrap required, opened a new log
I20260926 14:40:00.042397 24800 ts_tablet_manager.cc:1397] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260926 14:40:00.042523 24801 tablet_bootstrap.cc:492] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2: No bootstrap required, opened a new log
I20260926 14:40:00.042582 24801 ts_tablet_manager.cc:1397] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:40:00.042829 24800 raft_consensus.cc:348] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.042932 24800 raft_consensus.cc:374] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:00.042965 24800 raft_consensus.cc:729] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e21628c67d745619e80038e9cb14fe8, State: Initialized, Role: FOLLOWER
I20260926 14:40:00.042943 24801 raft_consensus.cc:348] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.043026 24801 raft_consensus.cc:374] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:00.043053 24801 raft_consensus.cc:729] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede771f9fef84a1a81651cf04d3dece2, State: Initialized, Role: FOLLOWER
I20260926 14:40:00.043079 24800 consensus_queue.cc:260] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.043165 24801 consensus_queue.cc:260] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.043262 24823 tablet_replica.cc:330] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c: stopping tablet replica
I20260926 14:40:00.043412 24801 ts_tablet_manager.cc:1428] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:40:00.043727 24800 ts_tablet_manager.cc:1428] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260926 14:40:00.044023 24823 raft_consensus.cc:2229] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:40:00.044375 24823 raft_consensus.cc:2258] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:40:00.044584 24799 tablet_bootstrap.cc:492] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c: Bootstrap starting.
I20260926 14:40:00.045504 24799 tablet_bootstrap.cc:654] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:00.046955 24799 tablet_bootstrap.cc:492] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c: No bootstrap required, opened a new log
I20260926 14:40:00.047036 24799 ts_tablet_manager.cc:1397] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:40:00.047398 24823 ts_tablet_manager.cc:1905] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:00.047472 24799 raft_consensus.cc:348] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.047573 24799 raft_consensus.cc:374] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:00.047601 24799 raft_consensus.cc:729] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273d2eea96f2470fb3abe9f63766735c, State: Initialized, Role: FOLLOWER
I20260926 14:40:00.047724 24799 consensus_queue.cc:260] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.047946 24799 ts_tablet_manager.cc:1428] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:40:00.050303 24823 ts_tablet_manager.cc:1918] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:40:00.050390 24823 log.cc:1196] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/900f75fd056a40b99f3ffac20e8ad158
I20260926 14:40:00.050628 24823 ts_tablet_manager.cc:1939] T 900f75fd056a40b99f3ffac20e8ad158 P 273d2eea96f2470fb3abe9f63766735c: Deleting consensus metadata
I20260926 14:40:00.051824 24270 catalog_manager.cc:4841] TS 273d2eea96f2470fb3abe9f63766735c (127.16.90.195:41457): tablet 900f75fd056a40b99f3ffac20e8ad158 (table test-table [id=6e74f6724e264b01b6d70836204a6998]) successfully deleted
I20260926 14:40:00.113178 24805 raft_consensus.cc:482] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:40:00.113322 24805 raft_consensus.cc:504] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.114380 24805 leader_election.cc:288] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331), 273d2eea96f2470fb3abe9f63766735c (127.16.90.195:41457)
I20260926 14:40:00.119524 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.114748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:34221 (local address 127.16.90.193:45331)
0926 14:40:00.114870 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.114874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.114992 (+ 118us) server_negotiation.cc:407] Connection header received
0926 14:40:00.115209 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.115212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.115251 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.115288 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:40:00.115712 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.116469 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.118110 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.118731 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.118832 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.118887 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.119297 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.119382 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:40:00.119930 24828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.114616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:45331 (local address 127.16.90.194:34221)
0926 14:40:00.114905 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:00.114919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:00.115082 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:00.115409 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:00.115412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:00.115424 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:40:00.115570 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.115578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.116597 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.116600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.117883 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.117896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.119005 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.119008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.119133 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.119157 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:40:00.119693 (+ 536us) client_negotiation.cc:240] Negotiation successful
0926 14:40:00.119774 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":63,"thread_start_us":66,"threads_started":1}
I20260926 14:40:00.120847 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71664f0db5047f794234182ee32d1a1" candidate_uuid: "ede771f9fef84a1a81651cf04d3dece2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e21628c67d745619e80038e9cb14fe8" is_pre_election: true
I20260926 14:40:00.121001 24443 raft_consensus.cc:2454] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ede771f9fef84a1a81651cf04d3dece2 in term 0.
I20260926 14:40:00.121340 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.115015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:43869 (local address 127.16.90.195:41457)
0926 14:40:00.115137 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.115141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.115155 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:40:00.115207 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.115210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.115254 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.115331 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:40:00.116620 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.117332 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.119257 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.120362 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.120490 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.120568 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.121062 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.121175 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:40:00.121837 24516 leader_election.cc:302] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e21628c67d745619e80038e9cb14fe8, ede771f9fef84a1a81651cf04d3dece2; no voters:
I20260926 14:40:00.121891 24827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.114663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:41457 (local address 127.16.90.194:43869)
0926 14:40:00.115023 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:00.115035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:00.115103 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:00.115358 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:00.115361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:00.115377 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:40:00.115574 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.115580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.117465 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.117469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.119065 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.119077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.120716 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.120720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.120870 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.120892 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:40:00.121584 (+ 692us) client_negotiation.cc:240] Negotiation successful
0926 14:40:00.121685 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":97,"thread_start_us":92,"threads_started":1}
I20260926 14:40:00.122246 24805 raft_consensus.cc:2781] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:40:00.122300 24805 raft_consensus.cc:482] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:40:00.122335 24805 raft_consensus.cc:3037] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:00.122733 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71664f0db5047f794234182ee32d1a1" candidate_uuid: "ede771f9fef84a1a81651cf04d3dece2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273d2eea96f2470fb3abe9f63766735c" is_pre_election: true
I20260926 14:40:00.123299 24725 raft_consensus.cc:2454] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ede771f9fef84a1a81651cf04d3dece2 in term 0.
I20260926 14:40:00.123605 24805 raft_consensus.cc:504] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.123999 24805 leader_election.cc:288] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [CANDIDATE]: Term 1 election: Requested vote from peers 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331), 273d2eea96f2470fb3abe9f63766735c (127.16.90.195:41457)
I20260926 14:40:00.124486 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71664f0db5047f794234182ee32d1a1" candidate_uuid: "ede771f9fef84a1a81651cf04d3dece2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e21628c67d745619e80038e9cb14fe8"
I20260926 14:40:00.124624 24443 raft_consensus.cc:3037] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:00.124588 24724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71664f0db5047f794234182ee32d1a1" candidate_uuid: "ede771f9fef84a1a81651cf04d3dece2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273d2eea96f2470fb3abe9f63766735c"
I20260926 14:40:00.124739 24724 raft_consensus.cc:3037] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:00.125743 24443 raft_consensus.cc:2454] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ede771f9fef84a1a81651cf04d3dece2 in term 1.
I20260926 14:40:00.126119 24516 leader_election.cc:302] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e21628c67d745619e80038e9cb14fe8, ede771f9fef84a1a81651cf04d3dece2; no voters:
I20260926 14:40:00.126314 24805 raft_consensus.cc:2781] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:40:00.126396 24724 raft_consensus.cc:2454] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ede771f9fef84a1a81651cf04d3dece2 in term 1.
I20260926 14:40:00.126495 24805 raft_consensus.cc:686] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 1 LEADER]: Becoming Leader. State: Replica: ede771f9fef84a1a81651cf04d3dece2, State: Running, Role: LEADER
I20260926 14:40:00.126735 24805 consensus_queue.cc:237] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.128935 24284 catalog_manager.cc:5495] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194). New cstate: current_term: 1 leader_uuid: "ede771f9fef84a1a81651cf04d3dece2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } health_report { overall_health: UNKNOWN } } }
Sep 26 14:40:00 dist-test-slave-x4sn krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433598, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.254@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
I20260926 14:40:00.181828 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.173527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.16.90.254:35447)
0926 14:40:00.173809 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.173824 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.173843 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:40:00.173912 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.173915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.173963 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.174140 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:00.174743 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.175470 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.176319 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.176642 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.176739 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.178878 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:00.178882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:00.178884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:40:00.178904 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:00.180306 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:00.180740 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:00.180742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:00.180743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:00.180815 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:00.181069 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:00.181070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:00.181071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:00.181233 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:00.181302 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.181538 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.181671 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":216,"thread_start_us":111,"threads_started":1}
I20260926 14:40:00.181829 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.173436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35447 (local address 127.0.0.1:41938)
0926 14:40:00.173717 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:00.173732 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:00.173834 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:00.174159 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:00.174161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:00.174527 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:00.174621 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.174627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.175578 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.175581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.176219 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.176228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.176752 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.176754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.176872 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.177401 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:00.177412 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:00.178758 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:00.180412 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:00.180422 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:00.180427 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:00.180660 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:00.180889 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:00.180891 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:00.180892 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:00.180973 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:00.181308 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:00.181312 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:00.181412 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 14:40:00.181496 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:40:00.181617 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":35,"thread_start_us":103,"threads_started":1}
I20260926 14:40:00.185469 24284 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:41938:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 14:40:00.186148 24284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:40:00.190447 24423 tablet_service.cc:1467] Processing CreateTablet for tablet 84ee5c8965ce4dd1bb62f176bdf851b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=742fbd5ce4054e29b02b62ca2f1019ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:40:00.190546 24565 tablet_service.cc:1467] Processing CreateTablet for tablet 84ee5c8965ce4dd1bb62f176bdf851b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=742fbd5ce4054e29b02b62ca2f1019ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:40:00.190635 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 84ee5c8965ce4dd1bb62f176bdf851b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=742fbd5ce4054e29b02b62ca2f1019ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:40:00.190846 24423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ee5c8965ce4dd1bb62f176bdf851b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:00.190953 24565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ee5c8965ce4dd1bb62f176bdf851b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:00.190953 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ee5c8965ce4dd1bb62f176bdf851b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:00.193415 24800 tablet_bootstrap.cc:492] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8: Bootstrap starting.
I20260926 14:40:00.193662 24799 tablet_bootstrap.cc:492] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: Bootstrap starting.
I20260926 14:40:00.194363 24800 tablet_bootstrap.cc:654] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:00.194387 24799 tablet_bootstrap.cc:654] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:00.194929 24801 tablet_bootstrap.cc:492] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2: Bootstrap starting.
I20260926 14:40:00.195729 24800 tablet_bootstrap.cc:492] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8: No bootstrap required, opened a new log
I20260926 14:40:00.195817 24800 ts_tablet_manager.cc:1397] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:40:00.195820 24801 tablet_bootstrap.cc:654] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:00.196205 24800 raft_consensus.cc:348] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.196302 24800 raft_consensus.cc:374] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:00.196323 24800 raft_consensus.cc:729] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e21628c67d745619e80038e9cb14fe8, State: Initialized, Role: FOLLOWER
I20260926 14:40:00.196426 24800 consensus_queue.cc:260] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.196708 24799 tablet_bootstrap.cc:492] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: No bootstrap required, opened a new log
I20260926 14:40:00.196686 24800 ts_tablet_manager.cc:1428] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:40:00.196683 24807 tablet_replica.cc:439] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } }
I20260926 14:40:00.196794 24799 ts_tablet_manager.cc:1397] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 14:40:00.196846 24807 tablet_replica.cc:442] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:40:00.197178 24799 raft_consensus.cc:348] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.197288 24799 raft_consensus.cc:374] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:00.197330 24799 raft_consensus.cc:729] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273d2eea96f2470fb3abe9f63766735c, State: Initialized, Role: FOLLOWER
I20260926 14:40:00.197445 24799 consensus_queue.cc:260] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.197651 24806 tablet_replica.cc:439] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } }
I20260926 14:40:00.197777 24806 tablet_replica.cc:442] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:40:00.197714 24799 ts_tablet_manager.cc:1428] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:40:00.198398 24801 tablet_bootstrap.cc:492] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2: No bootstrap required, opened a new log
I20260926 14:40:00.198477 24801 ts_tablet_manager.cc:1397] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 14:40:00.198894 24801 raft_consensus.cc:348] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.198992 24801 raft_consensus.cc:374] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:00.199025 24801 raft_consensus.cc:729] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede771f9fef84a1a81651cf04d3dece2, State: Initialized, Role: FOLLOWER
I20260926 14:40:00.199137 24801 consensus_queue.cc:260] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.199381 24801 ts_tablet_manager.cc:1428] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:40:00.199460 24805 tablet_replica.cc:439] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } }
I20260926 14:40:00.199579 24805 tablet_replica.cc:442] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:40:00.442623 24806 raft_consensus.cc:482] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:40:00.442759 24806 raft_consensus.cc:504] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.443113 24806 leader_election.cc:288] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331), ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194:41703)
I20260926 14:40:00.443573 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ee5c8965ce4dd1bb62f176bdf851b0" candidate_uuid: "273d2eea96f2470fb3abe9f63766735c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede771f9fef84a1a81651cf04d3dece2" is_pre_election: true
I20260926 14:40:00.443573 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ee5c8965ce4dd1bb62f176bdf851b0" candidate_uuid: "273d2eea96f2470fb3abe9f63766735c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e21628c67d745619e80038e9cb14fe8" is_pre_election: true
I20260926 14:40:00.443756 24443 raft_consensus.cc:2454] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 273d2eea96f2470fb3abe9f63766735c in term 0.
I20260926 14:40:00.443754 24585 raft_consensus.cc:2454] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 273d2eea96f2470fb3abe9f63766735c in term 0.
I20260926 14:40:00.444061 24658 leader_election.cc:302] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 273d2eea96f2470fb3abe9f63766735c, 2e21628c67d745619e80038e9cb14fe8, ede771f9fef84a1a81651cf04d3dece2; no voters:
I20260926 14:40:00.444267 24806 raft_consensus.cc:2781] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:40:00.444329 24806 raft_consensus.cc:482] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:40:00.444375 24806 raft_consensus.cc:3037] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:00.445429 24806 raft_consensus.cc:504] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.445737 24806 leader_election.cc:288] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [CANDIDATE]: Term 1 election: Requested vote from peers 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331), ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194:41703)
I20260926 14:40:00.446069 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ee5c8965ce4dd1bb62f176bdf851b0" candidate_uuid: "273d2eea96f2470fb3abe9f63766735c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede771f9fef84a1a81651cf04d3dece2"
I20260926 14:40:00.446112 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ee5c8965ce4dd1bb62f176bdf851b0" candidate_uuid: "273d2eea96f2470fb3abe9f63766735c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e21628c67d745619e80038e9cb14fe8"
I20260926 14:40:00.446199 24585 raft_consensus.cc:3037] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:00.446221 24443 raft_consensus.cc:3037] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:00.447017 24443 raft_consensus.cc:2454] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 273d2eea96f2470fb3abe9f63766735c in term 1.
I20260926 14:40:00.447093 24585 raft_consensus.cc:2454] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 273d2eea96f2470fb3abe9f63766735c in term 1.
I20260926 14:40:00.447292 24658 leader_election.cc:302] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 273d2eea96f2470fb3abe9f63766735c, 2e21628c67d745619e80038e9cb14fe8; no voters:
I20260926 14:40:00.447448 24806 raft_consensus.cc:2781] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:40:00.447532 24806 raft_consensus.cc:686] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [term 1 LEADER]: Becoming Leader. State: Replica: 273d2eea96f2470fb3abe9f63766735c, State: Running, Role: LEADER
I20260926 14:40:00.447685 24806 consensus_queue.cc:237] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.448308 24809 tablet_replica.cc:439] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: TxnStatusTablet state changed. Reason: New leader 273d2eea96f2470fb3abe9f63766735c. Latest consensus state: current_term: 1 leader_uuid: "273d2eea96f2470fb3abe9f63766735c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } }
I20260926 14:40:00.448413 24809 tablet_replica.cc:442] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:40:00.448630 24842 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 14:40:00.448825 24842 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 14:40:00.449350 24283 catalog_manager.cc:5495] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c reported cstate change: term changed from 0 to 1, leader changed from <none> to 273d2eea96f2470fb3abe9f63766735c (127.16.90.195). New cstate: current_term: 1 leader_uuid: "273d2eea96f2470fb3abe9f63766735c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } health_report { overall_health: HEALTHY } } }
I20260926 14:40:00.493014 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.489635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:41457 (local address 127.0.0.1:34024)
0926 14:40:00.489848 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:00.489863 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:00.489992 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:00.490268 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:00.490271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:00.490278 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:40:00.490379 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.490384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.491196 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.491200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.491917 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.491927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.492341 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.492345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.492434 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.492476 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.492742 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.492756 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 14:40:00.492816 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 14:40:00.492898 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 14:40:00.493014 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.489755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.16.90.195:41457)
0926 14:40:00.489890 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.489894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.489908 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:40:00.490120 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.490124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.490175 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.490235 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:40:00.490472 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.491104 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.492023 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.492247 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.492332 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.492599 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.492672 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.492729 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.492834 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.492883 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:40:00.495373 24585 raft_consensus.cc:1261] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Refusing update from remote peer 273d2eea96f2470fb3abe9f63766735c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:40:00.495440 24443 raft_consensus.cc:1261] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Refusing update from remote peer 273d2eea96f2470fb3abe9f63766735c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:40:00.495828 24806 consensus_queue.cc:1035] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:40:00.496207 24806 consensus_queue.cc:1035] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:40:00.497668 24807 tablet_replica.cc:439] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8: TxnStatusTablet state changed. Reason: New leader 273d2eea96f2470fb3abe9f63766735c. Latest consensus state: current_term: 1 leader_uuid: "273d2eea96f2470fb3abe9f63766735c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } }
I20260926 14:40:00.497790 24807 tablet_replica.cc:442] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:40:00.498150 24805 tablet_replica.cc:439] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2: TxnStatusTablet state changed. Reason: New leader 273d2eea96f2470fb3abe9f63766735c. Latest consensus state: current_term: 1 leader_uuid: "273d2eea96f2470fb3abe9f63766735c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } }
I20260926 14:40:00.498256 24805 tablet_replica.cc:442] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:40:00.500247 24806 tablet_replica.cc:439] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "273d2eea96f2470fb3abe9f63766735c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } }
I20260926 14:40:00.500342 24806 tablet_replica.cc:442] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:40:00.501467 24807 tablet_replica.cc:439] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "273d2eea96f2470fb3abe9f63766735c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } }
I20260926 14:40:00.501580 24807 tablet_replica.cc:442] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 2e21628c67d745619e80038e9cb14fe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:40:00.508648 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.504593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.16.90.194:41703)
0926 14:40:00.504900 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.504907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.505380 (+ 473us) server_negotiation.cc:407] Connection header received
0926 14:40:00.505576 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.505579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.505630 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.505730 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:40:00.505962 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.506463 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.507270 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.507504 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.507598 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.507863 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.508272 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.508373 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.508440 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.508492 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":90,"threads_started":1}
I20260926 14:40:00.512040 24809 tablet_replica.cc:439] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "273d2eea96f2470fb3abe9f63766735c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } }
I20260926 14:40:00.512164 24809 tablet_replica.cc:442] T 84ee5c8965ce4dd1bb62f176bdf851b0 P 273d2eea96f2470fb3abe9f63766735c: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:40:00.513339 24805 tablet_replica.cc:439] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "273d2eea96f2470fb3abe9f63766735c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } } }
I20260926 14:40:00.513440 24805 tablet_replica.cc:442] T 84ee5c8965ce4dd1bb62f176bdf851b0 P ede771f9fef84a1a81651cf04d3dece2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:40:00.518898 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.516146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:43135 (local address 127.16.90.195:41457)
0926 14:40:00.516244 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.516248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.516259 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:40:00.516296 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.516299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.516339 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.516395 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:40:00.516632 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.517113 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.517954 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.518173 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.518256 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.518518 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.518591 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.518645 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.518737 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.518781 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:40:00.518937 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.515838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:41457 (local address 127.16.90.194:43135)
0926 14:40:00.516089 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:00.516103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:00.516191 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:00.516418 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:00.516421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:00.516428 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:40:00.516533 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.516539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.517210 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.517213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.517843 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.517851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.518265 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.518269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.518360 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.518399 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.518664 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.518671 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 14:40:00.518727 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:40:00.518798 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260926 14:40:00.524960 24443 raft_consensus.cc:1261] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Refusing update from remote peer ede771f9fef84a1a81651cf04d3dece2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:40:00.525102 24724 raft_consensus.cc:1261] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Refusing update from remote peer ede771f9fef84a1a81651cf04d3dece2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:40:00.525516 24829 consensus_queue.cc:1035] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:40:00.525689 24805 consensus_queue.cc:1035] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:40:00.548329 24859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.544852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:41703 (local address 127.16.90.195:40645)
0926 14:40:00.545178 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:00.545194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:00.545326 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:00.545555 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:00.545558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:00.545565 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:40:00.545666 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.545671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.546414 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.546417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.547248 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.547257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.547686 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.547690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.547772 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.547814 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.548070 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.548075 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:40:00.548131 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:40:00.548207 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260926 14:40:00.548321 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.544888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:40645 (local address 127.16.90.194:41703)
0926 14:40:00.544990 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.544994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.545257 (+ 263us) server_negotiation.cc:407] Connection header received
0926 14:40:00.545409 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.545412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.545464 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.545534 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:40:00.545749 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.546314 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.547347 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.547574 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.547656 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.547928 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.547994 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.548049 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.548140 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.548187 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:40:00.570411 24283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:40:00.570603 24283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:40:00.572809 24283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a: Sending DeleteTablet for 3 replicas of tablet d71664f0db5047f794234182ee32d1a1
I20260926 14:40:00.573347 24423 tablet_service.cc:1514] Processing DeleteTablet for tablet d71664f0db5047f794234182ee32d1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:00 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:40578
I20260926 14:40:00.573484 24565 tablet_service.cc:1514] Processing DeleteTablet for tablet d71664f0db5047f794234182ee32d1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:00 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:38192
I20260926 14:40:00.573813 24862 tablet_replica.cc:330] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8: stopping tablet replica
I20260926 14:40:00.573987 24862 raft_consensus.cc:2229] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:40:00.574157 24862 raft_consensus.cc:2258] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:40:00.574295 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet d71664f0db5047f794234182ee32d1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:00 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:34004
I20260926 14:40:00.574594 24863 tablet_replica.cc:330] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2: stopping tablet replica
I20260926 14:40:00.574968 24862 ts_tablet_manager.cc:1905] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:00.576659 24863 raft_consensus.cc:2229] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:40:00.576985 24863 raft_consensus.cc:2258] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:40:00.577636 24863 ts_tablet_manager.cc:1905] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:00.578075 24862 ts_tablet_manager.cc:1918] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:40:00.578192 24862 log.cc:1196] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/wal/wals/d71664f0db5047f794234182ee32d1a1
I20260926 14:40:00.578510 24862 ts_tablet_manager.cc:1939] T d71664f0db5047f794234182ee32d1a1 P 2e21628c67d745619e80038e9cb14fe8: Deleting consensus metadata
I20260926 14:40:00.579181 24271 catalog_manager.cc:4841] TS 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331): tablet d71664f0db5047f794234182ee32d1a1 (table test-table [id=8ea66853c9674ad6832ce41c979c5bd1]) successfully deleted
I20260926 14:40:00.580382 24863 ts_tablet_manager.cc:1918] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:40:00.580464 24863 log.cc:1196] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/d71664f0db5047f794234182ee32d1a1
I20260926 14:40:00.580709 24863 ts_tablet_manager.cc:1939] T d71664f0db5047f794234182ee32d1a1 P ede771f9fef84a1a81651cf04d3dece2: Deleting consensus metadata
I20260926 14:40:00.581259 24864 tablet_replica.cc:330] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c: stopping tablet replica
I20260926 14:40:00.581359 24269 catalog_manager.cc:4841] TS ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194:41703): tablet d71664f0db5047f794234182ee32d1a1 (table test-table [id=8ea66853c9674ad6832ce41c979c5bd1]) successfully deleted
I20260926 14:40:00.581414 24864 raft_consensus.cc:2229] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:40:00.581615 24864 raft_consensus.cc:2258] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:40:00.582235 24864 ts_tablet_manager.cc:1905] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:00.585299 24864 ts_tablet_manager.cc:1918] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:40:00.585378 24864 log.cc:1196] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/d71664f0db5047f794234182ee32d1a1
I20260926 14:40:00.585613 24864 ts_tablet_manager.cc:1939] T d71664f0db5047f794234182ee32d1a1 P 273d2eea96f2470fb3abe9f63766735c: Deleting consensus metadata
I20260926 14:40:00.586398 24270 catalog_manager.cc:4841] TS 273d2eea96f2470fb3abe9f63766735c (127.16.90.195:41457): tablet d71664f0db5047f794234182ee32d1a1 (table test-table [id=8ea66853c9674ad6832ce41c979c5bd1]) successfully deleted
Sep 26 14:40:00 dist-test-slave-x4sn krb5kdc[24240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:40:00 dist-test-slave-x4sn krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:00 dist-test-slave-x4sn krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:40:00.613246 24837 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:00.606855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.16.90.254:35447)
0926 14:40:00.606988 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.606994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.607203 (+ 209us) server_negotiation.cc:407] Connection header received
0926 14:40:00.607334 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.607337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.607383 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.607450 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:00.608252 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.608821 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.609744 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.609979 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.610068 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.613119 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41942: BlockingRecv error: recv got EOF from 127.0.0.1:41942 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 14:40:00 dist-test-slave-x4sn krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM
I20260926 14:40:00.623126 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.614847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.16.90.254:35447)
0926 14:40:00.614953 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.614957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.615009 (+ 52us) server_negotiation.cc:407] Connection header received
0926 14:40:00.615154 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.615157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.615202 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.615285 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:00.615867 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.616647 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.617397 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.617728 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.617823 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.620119 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:00.620123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:00.620126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:40:00.620152 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:00.621554 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:00.621954 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:00.621957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:00.621959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:00.622044 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:00.622301 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:00.622304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:00.622306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:00.622506 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:00.622628 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.622846 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.622957 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:40:00.625506 24283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:40:00.625944 24283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:40:00.630398 24423 tablet_service.cc:1467] Processing CreateTablet for tablet 03940331ba8f4e9f9fa7b74c27764dde (DEFAULT_TABLE table=test-table [id=30fa8f895fb04c80a2e87778c937dcee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:40:00.630602 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 03940331ba8f4e9f9fa7b74c27764dde (DEFAULT_TABLE table=test-table [id=30fa8f895fb04c80a2e87778c937dcee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:40:00.630686 24423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03940331ba8f4e9f9fa7b74c27764dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:00.630868 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03940331ba8f4e9f9fa7b74c27764dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:00.632578 24800 tablet_bootstrap.cc:492] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8: Bootstrap starting.
I20260926 14:40:00.633226 24799 tablet_bootstrap.cc:492] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c: Bootstrap starting.
I20260926 14:40:00.633437 24800 tablet_bootstrap.cc:654] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:00.633441 24565 tablet_service.cc:1467] Processing CreateTablet for tablet 03940331ba8f4e9f9fa7b74c27764dde (DEFAULT_TABLE table=test-table [id=30fa8f895fb04c80a2e87778c937dcee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:40:00.633687 24565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03940331ba8f4e9f9fa7b74c27764dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:00.633950 24799 tablet_bootstrap.cc:654] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:00.634904 24799 tablet_bootstrap.cc:492] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c: No bootstrap required, opened a new log
I20260926 14:40:00.634904 24800 tablet_bootstrap.cc:492] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8: No bootstrap required, opened a new log
I20260926 14:40:00.634982 24799 ts_tablet_manager.cc:1397] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:40:00.634982 24800 ts_tablet_manager.cc:1397] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:40:00.635362 24800 raft_consensus.cc:348] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.635351 24799 raft_consensus.cc:348] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.635453 24799 raft_consensus.cc:374] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:00.635453 24800 raft_consensus.cc:374] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:00.635483 24799 raft_consensus.cc:729] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273d2eea96f2470fb3abe9f63766735c, State: Initialized, Role: FOLLOWER
I20260926 14:40:00.635493 24800 raft_consensus.cc:729] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e21628c67d745619e80038e9cb14fe8, State: Initialized, Role: FOLLOWER
I20260926 14:40:00.635610 24799 consensus_queue.cc:260] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.635610 24800 consensus_queue.cc:260] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.635905 24800 ts_tablet_manager.cc:1428] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:40:00.635905 24799 ts_tablet_manager.cc:1428] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:40:00.635960 24801 tablet_bootstrap.cc:492] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2: Bootstrap starting.
I20260926 14:40:00.636772 24801 tablet_bootstrap.cc:654] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:00.639684 24801 tablet_bootstrap.cc:492] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2: No bootstrap required, opened a new log
I20260926 14:40:00.639765 24801 ts_tablet_manager.cc:1397] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 14:40:00.640173 24801 raft_consensus.cc:348] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.640303 24801 raft_consensus.cc:374] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:00.640341 24801 raft_consensus.cc:729] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede771f9fef84a1a81651cf04d3dece2, State: Initialized, Role: FOLLOWER
I20260926 14:40:00.640453 24801 consensus_queue.cc:260] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.640704 24801 ts_tablet_manager.cc:1428] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:40:00.728925 24855 raft_consensus.cc:482] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:40:00.729071 24855 raft_consensus.cc:504] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.729501 24855 leader_election.cc:288] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331), 273d2eea96f2470fb3abe9f63766735c (127.16.90.195:41457)
I20260926 14:40:00.729940 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03940331ba8f4e9f9fa7b74c27764dde" candidate_uuid: "ede771f9fef84a1a81651cf04d3dece2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e21628c67d745619e80038e9cb14fe8" is_pre_election: true
I20260926 14:40:00.729944 24724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03940331ba8f4e9f9fa7b74c27764dde" candidate_uuid: "ede771f9fef84a1a81651cf04d3dece2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273d2eea96f2470fb3abe9f63766735c" is_pre_election: true
I20260926 14:40:00.730073 24443 raft_consensus.cc:2454] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ede771f9fef84a1a81651cf04d3dece2 in term 0.
I20260926 14:40:00.730090 24724 raft_consensus.cc:2454] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ede771f9fef84a1a81651cf04d3dece2 in term 0.
I20260926 14:40:00.730372 24516 leader_election.cc:302] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e21628c67d745619e80038e9cb14fe8, ede771f9fef84a1a81651cf04d3dece2; no voters:
I20260926 14:40:00.730547 24855 raft_consensus.cc:2781] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:40:00.730607 24855 raft_consensus.cc:482] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:40:00.730647 24855 raft_consensus.cc:3037] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:00.731600 24855 raft_consensus.cc:504] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.731941 24855 leader_election.cc:288] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [CANDIDATE]: Term 1 election: Requested vote from peers 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331), 273d2eea96f2470fb3abe9f63766735c (127.16.90.195:41457)
I20260926 14:40:00.732290 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03940331ba8f4e9f9fa7b74c27764dde" candidate_uuid: "ede771f9fef84a1a81651cf04d3dece2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e21628c67d745619e80038e9cb14fe8"
I20260926 14:40:00.732290 24724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03940331ba8f4e9f9fa7b74c27764dde" candidate_uuid: "ede771f9fef84a1a81651cf04d3dece2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273d2eea96f2470fb3abe9f63766735c"
I20260926 14:40:00.732409 24443 raft_consensus.cc:3037] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:00.732409 24724 raft_consensus.cc:3037] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:00.733501 24443 raft_consensus.cc:2454] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ede771f9fef84a1a81651cf04d3dece2 in term 1.
I20260926 14:40:00.733533 24724 raft_consensus.cc:2454] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ede771f9fef84a1a81651cf04d3dece2 in term 1.
I20260926 14:40:00.733791 24516 leader_election.cc:302] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e21628c67d745619e80038e9cb14fe8, ede771f9fef84a1a81651cf04d3dece2; no voters:
I20260926 14:40:00.733944 24855 raft_consensus.cc:2781] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:40:00.734026 24855 raft_consensus.cc:686] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 1 LEADER]: Becoming Leader. State: Replica: ede771f9fef84a1a81651cf04d3dece2, State: Running, Role: LEADER
I20260926 14:40:00.734148 24855 consensus_queue.cc:237] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.735637 24283 catalog_manager.cc:5495] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194). New cstate: current_term: 1 leader_uuid: "ede771f9fef84a1a81651cf04d3dece2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } health_report { overall_health: UNKNOWN } } }
I20260926 14:40:00.765581 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.762621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.16.90.194:41703)
0926 14:40:00.762758 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.762762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.762777 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:40:00.762907 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.762909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.762956 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.763031 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:40:00.763249 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.763803 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.764631 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.764846 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.764931 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.765171 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.765237 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.765292 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.765381 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.765444 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:40:00.767390 24724 raft_consensus.cc:1261] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Refusing update from remote peer ede771f9fef84a1a81651cf04d3dece2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:40:00.767390 24443 raft_consensus.cc:1261] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Refusing update from remote peer ede771f9fef84a1a81651cf04d3dece2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:40:00.767786 24855 consensus_queue.cc:1035] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:40:00.767944 24805 consensus_queue.cc:1035] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:40:00.774497 24284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:40:00.774641 24284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:40:00.776346 24284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a: Sending DeleteTablet for 3 replicas of tablet 03940331ba8f4e9f9fa7b74c27764dde
I20260926 14:40:00.776777 24423 tablet_service.cc:1514] Processing DeleteTablet for tablet 03940331ba8f4e9f9fa7b74c27764dde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:00 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:40578
I20260926 14:40:00.776907 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 03940331ba8f4e9f9fa7b74c27764dde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:00 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:34004
I20260926 14:40:00.776985 24862 tablet_replica.cc:330] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8: stopping tablet replica
I20260926 14:40:00.777091 24864 tablet_replica.cc:330] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c: stopping tablet replica
I20260926 14:40:00.777105 24862 raft_consensus.cc:2229] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:40:00.777208 24864 raft_consensus.cc:2229] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:40:00.777242 24862 raft_consensus.cc:2258] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:40:00.777278 24565 tablet_service.cc:1514] Processing DeleteTablet for tablet 03940331ba8f4e9f9fa7b74c27764dde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:00 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:38192
I20260926 14:40:00.777325 24864 raft_consensus.cc:2258] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:40:00.777421 24863 tablet_replica.cc:330] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2: stopping tablet replica
I20260926 14:40:00.777504 24863 raft_consensus.cc:2229] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:40:00.777487 24284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:40:00.777628 24863 raft_consensus.cc:2258] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:40:00.777801 24284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:40:00.778198 24864 ts_tablet_manager.cc:1905] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:00.778242 24863 ts_tablet_manager.cc:1905] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:00.778798 24862 ts_tablet_manager.cc:1905] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:00.780458 24864 ts_tablet_manager.cc:1918] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:40:00.780526 24864 log.cc:1196] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/03940331ba8f4e9f9fa7b74c27764dde
I20260926 14:40:00.780543 24863 ts_tablet_manager.cc:1918] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:40:00.780614 24863 log.cc:1196] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/03940331ba8f4e9f9fa7b74c27764dde
I20260926 14:40:00.780750 24864 ts_tablet_manager.cc:1939] T 03940331ba8f4e9f9fa7b74c27764dde P 273d2eea96f2470fb3abe9f63766735c: Deleting consensus metadata
I20260926 14:40:00.780856 24863 ts_tablet_manager.cc:1939] T 03940331ba8f4e9f9fa7b74c27764dde P ede771f9fef84a1a81651cf04d3dece2: Deleting consensus metadata
I20260926 14:40:00.780989 24862 ts_tablet_manager.cc:1918] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:40:00.781059 24862 log.cc:1196] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/wal/wals/03940331ba8f4e9f9fa7b74c27764dde
I20260926 14:40:00.781281 24862 ts_tablet_manager.cc:1939] T 03940331ba8f4e9f9fa7b74c27764dde P 2e21628c67d745619e80038e9cb14fe8: Deleting consensus metadata
I20260926 14:40:00.781388 24270 catalog_manager.cc:4841] TS 273d2eea96f2470fb3abe9f63766735c (127.16.90.195:41457): tablet 03940331ba8f4e9f9fa7b74c27764dde (table test-table [id=30fa8f895fb04c80a2e87778c937dcee]) successfully deleted
I20260926 14:40:00.782138 24271 catalog_manager.cc:4841] TS 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331): tablet 03940331ba8f4e9f9fa7b74c27764dde (table test-table [id=30fa8f895fb04c80a2e87778c937dcee]) successfully deleted
I20260926 14:40:00.782367 24269 catalog_manager.cc:4841] TS ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194:41703): tablet 03940331ba8f4e9f9fa7b74c27764dde (table test-table [id=30fa8f895fb04c80a2e87778c937dcee]) successfully deleted
I20260926 14:40:00.783607 24423 tablet_service.cc:1467] Processing CreateTablet for tablet 91c52fa1f46742d59137ae2c1f054b1d (DEFAULT_TABLE table=test-table [id=791cf829dd0d4101a7f38a45abbd8674]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:40:00.783708 24565 tablet_service.cc:1467] Processing CreateTablet for tablet 91c52fa1f46742d59137ae2c1f054b1d (DEFAULT_TABLE table=test-table [id=791cf829dd0d4101a7f38a45abbd8674]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:40:00.783748 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 91c52fa1f46742d59137ae2c1f054b1d (DEFAULT_TABLE table=test-table [id=791cf829dd0d4101a7f38a45abbd8674]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:40:00.783860 24423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c52fa1f46742d59137ae2c1f054b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:00.783897 24565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c52fa1f46742d59137ae2c1f054b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:00.783972 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c52fa1f46742d59137ae2c1f054b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:00.785677 24799 tablet_bootstrap.cc:492] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c: Bootstrap starting.
I20260926 14:40:00.785780 24800 tablet_bootstrap.cc:492] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8: Bootstrap starting.
I20260926 14:40:00.785837 24801 tablet_bootstrap.cc:492] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2: Bootstrap starting.
I20260926 14:40:00.786412 24801 tablet_bootstrap.cc:654] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:00.786445 24799 tablet_bootstrap.cc:654] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:00.786796 24800 tablet_bootstrap.cc:654] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:00.787771 24799 tablet_bootstrap.cc:492] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c: No bootstrap required, opened a new log
I20260926 14:40:00.787851 24799 ts_tablet_manager.cc:1397] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:40:00.788012 24801 tablet_bootstrap.cc:492] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2: No bootstrap required, opened a new log
I20260926 14:40:00.788074 24801 ts_tablet_manager.cc:1397] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:40:00.788172 24799 raft_consensus.cc:348] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.788249 24799 raft_consensus.cc:374] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:00.788267 24799 raft_consensus.cc:729] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273d2eea96f2470fb3abe9f63766735c, State: Initialized, Role: FOLLOWER
I20260926 14:40:00.788364 24799 consensus_queue.cc:260] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.788384 24801 raft_consensus.cc:348] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.788470 24801 raft_consensus.cc:374] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:00.788493 24801 raft_consensus.cc:729] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede771f9fef84a1a81651cf04d3dece2, State: Initialized, Role: FOLLOWER
I20260926 14:40:00.788611 24799 ts_tablet_manager.cc:1428] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:40:00.788589 24801 consensus_queue.cc:260] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.788816 24801 ts_tablet_manager.cc:1428] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:40:00.790156 24800 tablet_bootstrap.cc:492] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8: No bootstrap required, opened a new log
I20260926 14:40:00.790241 24800 ts_tablet_manager.cc:1397] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 14:40:00.790580 24800 raft_consensus.cc:348] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.790668 24800 raft_consensus.cc:374] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:00.790715 24800 raft_consensus.cc:729] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e21628c67d745619e80038e9cb14fe8, State: Initialized, Role: FOLLOWER
I20260926 14:40:00.790822 24800 consensus_queue.cc:260] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.791024 24800 ts_tablet_manager.cc:1428] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:40:00.910966 24807 raft_consensus.cc:482] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:40:00.911103 24807 raft_consensus.cc:504] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.912338 24807 leader_election.cc:288] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194:41703), 273d2eea96f2470fb3abe9f63766735c (127.16.90.195:41457)
I20260926 14:40:00.916406 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.912620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:43263 (local address 127.16.90.194:41703)
0926 14:40:00.912819 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.912824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.912901 (+ 77us) server_negotiation.cc:407] Connection header received
0926 14:40:00.913064 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.913067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.913120 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.913207 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:40:00.913542 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.914207 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.915378 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.915847 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.915940 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.915982 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.916196 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.916276 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 14:40:00.916420 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.912918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:46443 (local address 127.16.90.195:41457)
0926 14:40:00.913035 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.913040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.913233 (+ 193us) server_negotiation.cc:407] Connection header received
0926 14:40:00.913381 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.913385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.913429 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.913499 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:40:00.913869 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.914391 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.915499 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.915939 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.916050 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.916096 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.916248 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.916319 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:40:00.916414 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.912529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:41703 (local address 127.16.90.193:43263)
0926 14:40:00.912809 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:00.912824 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:00.912939 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:00.913230 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:00.913233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:00.913253 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:40:00.913395 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.913402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.914331 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.914335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.915261 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.915270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.915972 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.915976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.916076 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.916098 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:40:00.916166 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:40:00.916233 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":4,"spinlock_wait_cycles":1408,"thread_start_us":86,"threads_started":1}
I20260926 14:40:00.916424 24888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.912635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:41457 (local address 127.16.90.193:46443)
0926 14:40:00.913134 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:00.913147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:00.913280 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:00.913552 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:00.913556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:00.913577 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:40:00.913727 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.913736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.914527 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.914530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.915381 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.915391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.916060 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.916063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.916135 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.916155 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:40:00.916226 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:40:00.916285 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":63,"threads_started":1}
I20260926 14:40:00.916930 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c52fa1f46742d59137ae2c1f054b1d" candidate_uuid: "2e21628c67d745619e80038e9cb14fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede771f9fef84a1a81651cf04d3dece2" is_pre_election: true
I20260926 14:40:00.917088 24585 raft_consensus.cc:2454] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e21628c67d745619e80038e9cb14fe8 in term 0.
I20260926 14:40:00.917279 24724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c52fa1f46742d59137ae2c1f054b1d" candidate_uuid: "2e21628c67d745619e80038e9cb14fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273d2eea96f2470fb3abe9f63766735c" is_pre_election: true
I20260926 14:40:00.917431 24724 raft_consensus.cc:2454] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e21628c67d745619e80038e9cb14fe8 in term 0.
I20260926 14:40:00.917486 24374 leader_election.cc:302] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e21628c67d745619e80038e9cb14fe8, ede771f9fef84a1a81651cf04d3dece2; no voters:
I20260926 14:40:00.917752 24807 raft_consensus.cc:2781] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:40:00.917812 24807 raft_consensus.cc:482] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:40:00.917851 24807 raft_consensus.cc:3037] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:00.918772 24807 raft_consensus.cc:504] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.919078 24807 leader_election.cc:288] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [CANDIDATE]: Term 1 election: Requested vote from peers ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194:41703), 273d2eea96f2470fb3abe9f63766735c (127.16.90.195:41457)
I20260926 14:40:00.919387 24724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c52fa1f46742d59137ae2c1f054b1d" candidate_uuid: "2e21628c67d745619e80038e9cb14fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273d2eea96f2470fb3abe9f63766735c"
I20260926 14:40:00.919380 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c52fa1f46742d59137ae2c1f054b1d" candidate_uuid: "2e21628c67d745619e80038e9cb14fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede771f9fef84a1a81651cf04d3dece2"
I20260926 14:40:00.919507 24724 raft_consensus.cc:3037] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:00.919521 24585 raft_consensus.cc:3037] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:00.920578 24585 raft_consensus.cc:2454] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e21628c67d745619e80038e9cb14fe8 in term 1.
I20260926 14:40:00.920584 24724 raft_consensus.cc:2454] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e21628c67d745619e80038e9cb14fe8 in term 1.
I20260926 14:40:00.920830 24374 leader_election.cc:302] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e21628c67d745619e80038e9cb14fe8, ede771f9fef84a1a81651cf04d3dece2; no voters:
I20260926 14:40:00.921042 24807 raft_consensus.cc:2781] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:40:00.921200 24807 raft_consensus.cc:686] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 1 LEADER]: Becoming Leader. State: Replica: 2e21628c67d745619e80038e9cb14fe8, State: Running, Role: LEADER
I20260926 14:40:00.921416 24807 consensus_queue.cc:237] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } }
I20260926 14:40:00.923875 24284 catalog_manager.cc:5495] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193). New cstate: current_term: 1 leader_uuid: "2e21628c67d745619e80038e9cb14fe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e21628c67d745619e80038e9cb14fe8" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 45331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 } health_report { overall_health: UNKNOWN } } }
I20260926 14:40:00.950830 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.947203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.16.90.193:45331)
0926 14:40:00.947494 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.947500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.947511 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:40:00.947581 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.947584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.947635 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.947776 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:40:00.947999 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.948574 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.949430 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.949677 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.949770 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.949980 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.950459 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.950538 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.950619 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.950666 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":113,"threads_started":1}
I20260926 14:40:00.960776 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.957662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:57997 (local address 127.16.90.195:41457)
0926 14:40:00.957829 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.957833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.957852 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:40:00.958008 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.958011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.958053 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.958113 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:40:00.958369 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.958866 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.959768 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.959988 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.960073 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.960354 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.960425 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.960482 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.960572 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.960636 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 14:40:00.960790 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.957482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:41457 (local address 127.16.90.193:57997)
0926 14:40:00.957792 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:00.957806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:00.957903 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:00.958138 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:00.958141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:00.958149 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:40:00.958269 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.958274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.958962 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.958966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.959624 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.959633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.960102 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.960106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.960190 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.960232 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.960495 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.960501 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:40:00.960557 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:40:00.960636 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":107,"threads_started":1}
I20260926 14:40:00.966511 24585 raft_consensus.cc:1261] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Refusing update from remote peer 2e21628c67d745619e80038e9cb14fe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:40:00.966572 24724 raft_consensus.cc:1261] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Refusing update from remote peer 2e21628c67d745619e80038e9cb14fe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:40:00.966969 24889 consensus_queue.cc:1035] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede771f9fef84a1a81651cf04d3dece2" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:40:00.967141 24807 consensus_queue.cc:1035] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "273d2eea96f2470fb3abe9f63766735c" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 41457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:40:00.983496 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.980710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:51871 (local address 127.16.90.193:45331)
0926 14:40:00.980806 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:00.980810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:00.980823 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:40:00.980953 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:00.980956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:00.981002 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:00.981059 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:40:00.981262 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.981775 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.982563 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.982787 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.982872 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.983087 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.983159 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.983221 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:00.983306 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 14:40:00.983356 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 14:40:00.983496 24859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:00.980640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:45331 (local address 127.16.90.195:51871)
0926 14:40:00.980770 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:00.980785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:00.980864 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:00.981074 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:00.981077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:00.981083 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:40:00.981175 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.981181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.981876 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.981880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.982461 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:00.982468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:00.982877 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:00.982881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:00.982964 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:00.982992 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:40:00.983235 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:40:00.983241 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:40:00.983297 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:40:00.983383 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 14:40:01.002729 24285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:40:01.002921 24285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:40:01.005074 24285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e063eace0a7149628b4caf51284e9d9a: Sending DeleteTablet for 3 replicas of tablet 91c52fa1f46742d59137ae2c1f054b1d
I20260926 14:40:01.005738 24423 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c52fa1f46742d59137ae2c1f054b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:00 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:40578
I20260926 14:40:01.005961 24565 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c52fa1f46742d59137ae2c1f054b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:00 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:38192
I20260926 14:40:01.006100 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c52fa1f46742d59137ae2c1f054b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:00 UTC) from {username='oryx', principal='oryx/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:34004
I20260926 14:40:01.006146 24862 tablet_replica.cc:330] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8: stopping tablet replica
I20260926 14:40:01.006268 24862 raft_consensus.cc:2229] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:40:01.006313 24864 tablet_replica.cc:330] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c: stopping tablet replica
I20260926 14:40:01.006441 24864 raft_consensus.cc:2229] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:40:01.006556 24862 raft_consensus.cc:2258] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:40:01.006567 24864 raft_consensus.cc:2258] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:40:01.006752 24863 tablet_replica.cc:330] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2: stopping tablet replica
I20260926 14:40:01.006882 24863 raft_consensus.cc:2229] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:40:01.006986 24863 raft_consensus.cc:2258] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:40:01.007534 24864 ts_tablet_manager.cc:1905] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:01.007654 24862 ts_tablet_manager.cc:1905] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:01.009631 24862 ts_tablet_manager.cc:1918] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:40:01.009706 24862 log.cc:1196] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-0/wal/wals/91c52fa1f46742d59137ae2c1f054b1d
I20260926 14:40:01.009912 24862 ts_tablet_manager.cc:1939] T 91c52fa1f46742d59137ae2c1f054b1d P 2e21628c67d745619e80038e9cb14fe8: Deleting consensus metadata
I20260926 14:40:01.010210 24864 ts_tablet_manager.cc:1918] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:40:01.010295 24864 log.cc:1196] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/91c52fa1f46742d59137ae2c1f054b1d
I20260926 14:40:01.010522 24864 ts_tablet_manager.cc:1939] T 91c52fa1f46742d59137ae2c1f054b1d P 273d2eea96f2470fb3abe9f63766735c: Deleting consensus metadata
I20260926 14:40:01.010670 24271 catalog_manager.cc:4841] TS 2e21628c67d745619e80038e9cb14fe8 (127.16.90.193:45331): tablet 91c52fa1f46742d59137ae2c1f054b1d (table test-table [id=791cf829dd0d4101a7f38a45abbd8674]) successfully deleted
I20260926 14:40:01.010821 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 24358
I20260926 14:40:01.011021 24863 ts_tablet_manager.cc:1905] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:01.013361 24863 ts_tablet_manager.cc:1918] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:40:01.013437 24863 log.cc:1196] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/91c52fa1f46742d59137ae2c1f054b1d
I20260926 14:40:01.013633 24270 catalog_manager.cc:4841] TS 273d2eea96f2470fb3abe9f63766735c (127.16.90.195:41457): tablet 91c52fa1f46742d59137ae2c1f054b1d (table test-table [id=791cf829dd0d4101a7f38a45abbd8674]) successfully deleted
I20260926 14:40:01.013685 24863 ts_tablet_manager.cc:1939] T 91c52fa1f46742d59137ae2c1f054b1d P ede771f9fef84a1a81651cf04d3dece2: Deleting consensus metadata
I20260926 14:40:01.014309 24269 catalog_manager.cc:4841] TS ede771f9fef84a1a81651cf04d3dece2 (127.16.90.194:41703): tablet 91c52fa1f46742d59137ae2c1f054b1d (table test-table [id=791cf829dd0d4101a7f38a45abbd8674]) successfully deleted
I20260926 14:40:01.018258 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 24498
I20260926 14:40:01.023888 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 24640
I20260926 14:40:01.029500 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 24253
2026-09-26T14:40:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3480 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:40:01 dist-test-slave-x4sn krb5kdc[24901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:40:01 dist-test-slave-x4sn krb5kdc[24901](info): set up 2 sockets
Sep 26 14:40:01 dist-test-slave-x4sn krb5kdc[24901](info): commencing operation
krb5kdc: starting...
W20260926 14:40:01.986142 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:40:02 dist-test-slave-x4sn krb5kdc[24901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:40:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:40:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:40:02.096007 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35967
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:35967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:40:02.205453 24914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:02.205747 24914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:02.205804 24914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:02.209888 24914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:40:02.209959 24914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:02.209986 24914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:02.210008 24914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:40:02.210032 24914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:40:02.215335 24914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:35967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:02.216405 24914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:02.217417 24914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:02.225730 24919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:02.225739 24922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:02.225795 24914 server_base.cc:1034] running on GCE node
W20260926 14:40:02.225741 24920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:02.226588 24914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:02.227715 24914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:02.228879 24914 hybrid_clock.cc:648] HybridClock initialized: now 1790433602228857 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T14:40:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1203 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:40:02 dist-test-slave-x4sn krb5kdc[24927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:40:02 dist-test-slave-x4sn krb5kdc[24927](info): set up 2 sockets
Sep 26 14:40:02 dist-test-slave-x4sn krb5kdc[24927](info): commencing operation
krb5kdc: starting...
W20260926 14:40:03.189592 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:40:03 dist-test-slave-x4sn krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:40:03.304004 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:34755
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:34755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:40:03.409008 24940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:03.409271 24940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:03.409327 24940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:03.413235 24940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:40:03.413288 24940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:03.413313 24940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:03.413333 24940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:40:03.413353 24940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:40:03.418434 24940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:34755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:03.419435 24940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:03.420439 24940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:03.428285 24948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:03.428311 24946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:03.428319 24945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:03.428542 24940 server_base.cc:1034] running on GCE node
I20260926 14:40:03.429163 24940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:03.430231 24940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:03.431384 24940 hybrid_clock.cc:648] HybridClock initialized: now 1790433603431363 us; error 34 us; skew 500 ppm
Sep 26 14:40:03 dist-test-slave-x4sn krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433603, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:40:03.439877 24940 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:40:03.440894 24940 webserver.cc:460] Webserver started at http://127.16.90.254:42831/ using document root <none> and password file <none>
I20260926 14:40:03.441462 24940 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:03.441538 24940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:03.441771 24940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:03.443897 24940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "9afb97a5fda64f24aecd04d8392b0954"
format_stamp: "Formatted at 2026-09-26 14:40:03 on dist-test-slave-x4sn"
server_key: "c2ab4970a80ebe3a69486d8f6075179f"
server_key_iv: "a8b5124100b92d262e62c3b21c9158a8"
server_key_version: "encryptionkey@0"
I20260926 14:40:03.444499 24940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "9afb97a5fda64f24aecd04d8392b0954"
format_stamp: "Formatted at 2026-09-26 14:40:03 on dist-test-slave-x4sn"
server_key: "c2ab4970a80ebe3a69486d8f6075179f"
server_key_iv: "a8b5124100b92d262e62c3b21c9158a8"
server_key_version: "encryptionkey@0"
I20260926 14:40:03.448167 24940 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:40:03.450438 24956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:03.451625 24940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:40:03.451787 24940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "9afb97a5fda64f24aecd04d8392b0954"
format_stamp: "Formatted at 2026-09-26 14:40:03 on dist-test-slave-x4sn"
server_key: "c2ab4970a80ebe3a69486d8f6075179f"
server_key_iv: "a8b5124100b92d262e62c3b21c9158a8"
server_key_version: "encryptionkey@0"
I20260926 14:40:03.451879 24940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:03.466261 24940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:03.468875 24940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:03.469043 24940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:03.475683 24940 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:34755
I20260926 14:40:03.475706 25008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:34755 every 8 connection(s)
I20260926 14:40:03.476673 24940 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:40:03.479485 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:03.479503 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 24940
I20260926 14:40:03.479624 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:40:03.479933 16747 external_mini_cluster.cc:1428] Setting key e881635a82249410436247a54a5f3db5
I20260926 14:40:03.485888 25009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: Bootstrap starting.
I20260926 14:40:03.488268 25009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: Neither blocks nor log segments found. Creating new log.
Sep 26 14:40:03 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:40:03.489140 25009 log.cc:824] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: Log is configured to *not* fsync() on all Append() calls
I20260926 14:40:03.491005 25009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: No bootstrap required, opened a new log
I20260926 14:40:03.493856 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:03.481287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59876 (local address 127.16.90.254:34755)
0926 14:40:03.481741 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:03.481756 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:03.481792 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:40:03.482733 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:03.482754 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:03.483124 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:03.483457 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:03.484294 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:03.485273 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:03.486134 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:03.486615 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:03.486765 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:03.489385 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:03.489390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:03.489403 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:40:03.489431 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:03.490997 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:03.491418 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:03.491423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:03.491428 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:03.491624 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:03.491949 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:03.491953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:03.491955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:03.492362 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:03.492535 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:03.492824 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 14:40:03.493110 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":126,"threads_started":1}
I20260926 14:40:03.494673 25009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afb97a5fda64f24aecd04d8392b0954" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 34755 } }
I20260926 14:40:03.494940 25009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:03.495007 25009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afb97a5fda64f24aecd04d8392b0954, State: Initialized, Role: FOLLOWER
I20260926 14:40:03.495483 25009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afb97a5fda64f24aecd04d8392b0954" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 34755 } }
I20260926 14:40:03.495635 25009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:40:03.495759 25009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:40:03.495860 25009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:03.496766 25009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afb97a5fda64f24aecd04d8392b0954" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 34755 } }
I20260926 14:40:03.497092 25009 leader_election.cc:302] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9afb97a5fda64f24aecd04d8392b0954; no voters:
I20260926 14:40:03.497375 25009 leader_election.cc:288] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:40:03.497506 25014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:40:03.497722 25014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 1 LEADER]: Becoming Leader. State: Replica: 9afb97a5fda64f24aecd04d8392b0954, State: Running, Role: LEADER
I20260926 14:40:03.498030 25014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afb97a5fda64f24aecd04d8392b0954" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 34755 } }
I20260926 14:40:03.498507 25009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:40:03.499619 25016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9afb97a5fda64f24aecd04d8392b0954. Latest consensus state: current_term: 1 leader_uuid: "9afb97a5fda64f24aecd04d8392b0954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afb97a5fda64f24aecd04d8392b0954" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 34755 } } }
I20260926 14:40:03.499606 25015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9afb97a5fda64f24aecd04d8392b0954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afb97a5fda64f24aecd04d8392b0954" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 34755 } } }
I20260926 14:40:03.499775 25016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [sys.catalog]: This master's current role is: LEADER
I20260926 14:40:03.499775 25015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [sys.catalog]: This master's current role is: LEADER
I20260926 14:40:03.500162 25023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:40:03.503110 25023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:40:03.508854 25023 catalog_manager.cc:1347] Generated new cluster ID: 204d37aac9734ab39b8334cf0d1ca700
I20260926 14:40:03.508984 25023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:40:03.522955 25023 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:40:03.524051 25023 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:40:03.530543 25023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: Generated new TSK 0
I20260926 14:40:03.531219 25023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:40:03.566531 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:34755
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:40:03.674052 25035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:03.674335 25035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:03.674393 25035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:03.678426 25035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:03.678498 25035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:03.678579 25035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:40:03.683924 25035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:34755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:03.684979 25035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:03.685997 25035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:03.695113 25040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:03.695145 25041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:03.695202 25035 server_base.cc:1034] running on GCE node
W20260926 14:40:03.695150 25043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:03.695641 25035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:03.696245 25035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:03.697405 25035 hybrid_clock.cc:648] HybridClock initialized: now 1790433603697387 us; error 34 us; skew 500 ppm
Sep 26 14:40:03 dist-test-slave-x4sn krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433603, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:40:03.700209 25035 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:40:03.701095 25035 webserver.cc:460] Webserver started at http://127.16.90.193:41277/ using document root <none> and password file <none>
I20260926 14:40:03.701629 25035 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:03.701690 25035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:03.701894 25035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:03.703943 25035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "b7b417fa63454a01871668aa4738613e"
format_stamp: "Formatted at 2026-09-26 14:40:03 on dist-test-slave-x4sn"
server_key: "c59afcc492881e17d2b24b5d51da4323"
server_key_iv: "15010da740dc3e0d9836a9fa47c2b6a7"
server_key_version: "encryptionkey@0"
I20260926 14:40:03.704459 25035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "b7b417fa63454a01871668aa4738613e"
format_stamp: "Formatted at 2026-09-26 14:40:03 on dist-test-slave-x4sn"
server_key: "c59afcc492881e17d2b24b5d51da4323"
server_key_iv: "15010da740dc3e0d9836a9fa47c2b6a7"
server_key_version: "encryptionkey@0"
I20260926 14:40:03.708102 25035 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:40:03.710378 25050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:03.711637 25035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:40:03.711772 25035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "b7b417fa63454a01871668aa4738613e"
format_stamp: "Formatted at 2026-09-26 14:40:03 on dist-test-slave-x4sn"
server_key: "c59afcc492881e17d2b24b5d51da4323"
server_key_iv: "15010da740dc3e0d9836a9fa47c2b6a7"
server_key_version: "encryptionkey@0"
I20260926 14:40:03.711862 25035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:03.732594 25035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:03.735610 25035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:03.735809 25035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:03.736385 25035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:03.737175 25035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:03.737224 25035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:03.737267 25035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:03.737295 25035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:03.745321 25035 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:43247
I20260926 14:40:03.745366 25164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:43247 every 8 connection(s)
I20260926 14:40:03.746169 25035 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:40:03.753050 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 25035
I20260926 14:40:03.753183 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:40:03.753496 16747 external_mini_cluster.cc:1428] Setting key efb0d6eeb8a2343df89861777bf06909
Sep 26 14:40:03 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433603, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:40:03.759518 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:03.747892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:35065 (local address 127.16.90.254:34755)
0926 14:40:03.748008 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:03.748012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:03.749279 (+ 1267us) server_negotiation.cc:407] Connection header received
0926 14:40:03.750087 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:03.750090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:03.750138 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:03.750212 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:03.751310 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:03.751914 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:03.753271 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:03.753536 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:03.753643 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:03.756233 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:03.756237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:03.756239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:40:03.756267 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:03.757579 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:03.758058 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:03.758061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:03.758063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:03.758144 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:03.758480 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:03.758484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:03.758485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:03.758665 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:03.758753 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:03.759225 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 14:40:03.759358 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:40:03.760310 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:03.748293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:35065)
0926 14:40:03.749121 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:03.749162 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:03.749909 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:03.750355 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:03.750362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:03.750702 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:03.751153 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:03.751166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:03.752069 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:03.752072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:03.753125 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:03.753137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:03.753694 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:03.753699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:03.753887 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:03.754458 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:03.754474 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:03.756071 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:03.757688 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:03.757693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:03.757704 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:03.757958 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:03.758259 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:03.758262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:03.758264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:03.758393 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:03.758765 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:03.758771 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:03.758998 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 14:40:03.759159 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 14:40:03.759476 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":517,"threads_started":1}
I20260926 14:40:03.761579 25165 heartbeater.cc:347] Connected to a master server at 127.16.90.254:34755
I20260926 14:40:03.761811 25165 heartbeater.cc:464] Registering TS with master...
I20260926 14:40:03.762264 25165 heartbeater.cc:511] Master 127.16.90.254:34755 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
I20260926 14:40:03.763653 24973 ts_manager.cc:194] Registered new tserver with Master: b7b417fa63454a01871668aa4738613e (127.16.90.193:43247)
I20260926 14:40:03.764758 24973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:35065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:40:03.778946 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:34755
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:40:03.886439 25170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:03.886680 25170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:03.886721 25170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:03.890684 25170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:03.890741 25170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:03.890797 25170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:40:03.895999 25170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:34755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:03.896907 25170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:03.897822 25170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:03.906915 25178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:03.906991 25176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:03.907032 25175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:03.907084 25170 server_base.cc:1034] running on GCE node
I20260926 14:40:03.907619 25170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:03.908252 25170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:03.909420 25170 hybrid_clock.cc:648] HybridClock initialized: now 1790433603909399 us; error 35 us; skew 500 ppm
Sep 26 14:40:03 dist-test-slave-x4sn krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433603, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:40:03.912242 25170 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:40:03.913051 25170 webserver.cc:460] Webserver started at http://127.16.90.194:42925/ using document root <none> and password file <none>
I20260926 14:40:03.913565 25170 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:03.913637 25170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:03.913834 25170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:03.915879 25170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "186550b7fad84412a6ff02d8d9bcdd5a"
format_stamp: "Formatted at 2026-09-26 14:40:03 on dist-test-slave-x4sn"
server_key: "5a81572bc1430cfc990dbfe69f8af99a"
server_key_iv: "c020118c3cb8f561e9f9777ea8a35a3d"
server_key_version: "encryptionkey@0"
I20260926 14:40:03.916388 25170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "186550b7fad84412a6ff02d8d9bcdd5a"
format_stamp: "Formatted at 2026-09-26 14:40:03 on dist-test-slave-x4sn"
server_key: "5a81572bc1430cfc990dbfe69f8af99a"
server_key_iv: "c020118c3cb8f561e9f9777ea8a35a3d"
server_key_version: "encryptionkey@0"
I20260926 14:40:03.919807 25170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:40:03.922144 25185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:03.923276 25170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:40:03.923379 25170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "186550b7fad84412a6ff02d8d9bcdd5a"
format_stamp: "Formatted at 2026-09-26 14:40:03 on dist-test-slave-x4sn"
server_key: "5a81572bc1430cfc990dbfe69f8af99a"
server_key_iv: "c020118c3cb8f561e9f9777ea8a35a3d"
server_key_version: "encryptionkey@0"
I20260926 14:40:03.923470 25170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:03.944535 25170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:03.948359 25170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:03.948549 25170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:03.949268 25170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:03.950209 25170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:03.950265 25170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:03.950317 25170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:03.950346 25170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:03.960013 25170 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:40167
I20260926 14:40:03.960134 25299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:40167 every 8 connection(s)
I20260926 14:40:03.960867 25170 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:40:03.964758 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 25170
I20260926 14:40:03.964864 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:40:03.965119 16747 external_mini_cluster.cc:1428] Setting key 70ab7d01eb6926d6b32795ccb5a0d3b0
Sep 26 14:40:03 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433603, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
I20260926 14:40:03.975356 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:03.962764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:51779 (local address 127.16.90.254:34755)
0926 14:40:03.962918 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:03.962923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:03.963837 (+ 914us) server_negotiation.cc:407] Connection header received
0926 14:40:03.965519 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:03.965523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:03.965581 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:03.965636 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:03.967104 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:03.967844 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:03.969371 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:03.969741 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:03.969841 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:03.972311 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:03.972315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:03.972318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:40:03.972340 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:03.973570 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:03.974072 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:03.974075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:03.974076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:03.974149 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:03.974441 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:03.974444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:03.974446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:03.974612 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:03.974704 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:03.975142 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 14:40:03.975237 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
I20260926 14:40:03.976431 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:03.963141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:51779)
0926 14:40:03.963638 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:03.963730 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:03.965270 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:03.966058 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:03.966068 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:03.966507 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:03.966967 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:03.966980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:03.967966 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:03.967970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:03.969263 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:03.969273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:03.969862 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:03.969865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:03.969999 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:03.970511 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:03.970527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:03.972131 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:03.973675 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:03.973680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:03.973695 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:03.973976 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:03.974227 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:03.974230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:03.974232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:03.974361 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:03.974725 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:03.974730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:03.974950 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 14:40:03.975104 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 14:40:03.975407 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":107,"threads_started":1}
I20260926 14:40:03.977488 25300 heartbeater.cc:347] Connected to a master server at 127.16.90.254:34755
I20260926 14:40:03.977711 25300 heartbeater.cc:464] Registering TS with master...
I20260926 14:40:03.978153 25300 heartbeater.cc:511] Master 127.16.90.254:34755 requested a full tablet report, sending...
I20260926 14:40:03.979081 24973 ts_manager.cc:194] Registered new tserver with Master: 186550b7fad84412a6ff02d8d9bcdd5a (127.16.90.194:40167)
I20260926 14:40:03.979621 24973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:51779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:40:03.990774 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:34755
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:40:04.096791 25305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:04.097041 25305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:04.097082 25305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:04.101233 25305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:04.101289 25305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:04.101357 25305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:40:04.106611 25305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:34755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:04.107638 25305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:04.108673 25305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:04.117785 25311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:04.117797 25310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:04.118053 25305 server_base.cc:1034] running on GCE node
W20260926 14:40:04.118273 25313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:04.118706 25305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:04.119410 25305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:04.122676 25305 hybrid_clock.cc:648] HybridClock initialized: now 1790433604122643 us; error 37 us; skew 500 ppm
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433604, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:40:04.125622 25305 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:40:04.126478 25305 webserver.cc:460] Webserver started at http://127.16.90.195:33115/ using document root <none> and password file <none>
I20260926 14:40:04.127007 25305 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:04.127081 25305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:04.127288 25305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:04.129410 25305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "2cec0cc04daf49eaab7de5551b2bf010"
format_stamp: "Formatted at 2026-09-26 14:40:04 on dist-test-slave-x4sn"
server_key: "2ca98d8b68cf2aea53f68d19e116373e"
server_key_iv: "b02cd4c813d7b04b39249ed1e6616171"
server_key_version: "encryptionkey@0"
I20260926 14:40:04.129911 25305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "2cec0cc04daf49eaab7de5551b2bf010"
format_stamp: "Formatted at 2026-09-26 14:40:04 on dist-test-slave-x4sn"
server_key: "2ca98d8b68cf2aea53f68d19e116373e"
server_key_iv: "b02cd4c813d7b04b39249ed1e6616171"
server_key_version: "encryptionkey@0"
I20260926 14:40:04.133460 25305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 14:40:04.136013 25320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:04.137136 25305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:40:04.137241 25305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "2cec0cc04daf49eaab7de5551b2bf010"
format_stamp: "Formatted at 2026-09-26 14:40:04 on dist-test-slave-x4sn"
server_key: "2ca98d8b68cf2aea53f68d19e116373e"
server_key_iv: "b02cd4c813d7b04b39249ed1e6616171"
server_key_version: "encryptionkey@0"
I20260926 14:40:04.137327 25305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:04.156505 25305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:04.159113 25305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:04.159276 25305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:04.159837 25305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:04.160600 25305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:04.160646 25305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:04.160694 25305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:04.160722 25305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:04.169055 25305 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:41049
I20260926 14:40:04.169085 25434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:41049 every 8 connection(s)
I20260926 14:40:04.169873 25305 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:40:04.176674 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 25305
I20260926 14:40:04.176808 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:40:04.177142 16747 external_mini_cluster.cc:1428] Setting key 0683a7a142e500c079dca733cb3c1d14
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433604, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:40:04.182376 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:04.171567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:50041 (local address 127.16.90.254:34755)
0926 14:40:04.171739 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:04.171743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:04.172657 (+ 914us) server_negotiation.cc:407] Connection header received
0926 14:40:04.173485 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:04.173487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:04.173532 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:04.173606 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:04.174733 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.175297 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.176378 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.176598 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.176687 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.179244 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:04.179248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:04.179251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:40:04.179278 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:04.180537 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:04.180991 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:04.180994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:04.180995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:04.181077 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:04.181436 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:04.181438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:04.181439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:04.181603 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:04.181672 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:04.182140 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 14:40:04.182263 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:40:04.183207 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:04.171939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:50041)
0926 14:40:04.172539 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:04.172577 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:04.173307 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:04.173802 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:04.173812 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:04.174145 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:04.174591 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.174605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.175391 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.175393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.176274 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.176283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.176769 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.176774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.176964 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.177546 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:04.177561 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:04.179094 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:04.180636 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:04.180641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:04.180651 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:04.180886 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:04.181195 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:04.181198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:04.181200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:04.181355 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:04.181687 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:04.181692 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:04.181932 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 14:40:04.182115 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 14:40:04.182363 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":116,"threads_started":1}
I20260926 14:40:04.184340 25435 heartbeater.cc:347] Connected to a master server at 127.16.90.254:34755
I20260926 14:40:04.184573 25435 heartbeater.cc:464] Registering TS with master...
I20260926 14:40:04.185024 25435 heartbeater.cc:511] Master 127.16.90.254:34755 requested a full tablet report, sending...
I20260926 14:40:04.185973 24973 ts_manager.cc:194] Registered new tserver with Master: 2cec0cc04daf49eaab7de5551b2bf010 (127.16.90.195:41049)
I20260926 14:40:04.186525 24973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:50041
I20260926 14:40:04.190739 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab.
Entry for principal oryx/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab.
WARNING: no policy specified for oryx/127.16.90.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab.
Entry for principal oryx/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab.
WARNING: no policy specified for oryx/127.16.90.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.90.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab.
Entry for principal oryx/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab.
I20260926 14:40:04.252178 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 25035
I20260926 14:40:04.257171 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 25170
I20260926 14:40:04.261843 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 25305
I20260926 14:40:04.266389 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 24940
I20260926 14:40:04.271979 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:34755
--webserver_interface=127.16.90.254
--webserver_port=42831
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:34755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:40:04.377475 25444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:04.377753 25444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:04.377808 25444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:04.381865 25444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:40:04.381932 25444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:04.381959 25444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:04.381980 25444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:40:04.382009 25444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:40:04.387272 25444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:34755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:04.388362 25444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:04.389360 25444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:04.397650 25450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:04.397693 25449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:04.397693 25452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:04.397913 25444 server_base.cc:1034] running on GCE node
I20260926 14:40:04.398521 25444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:04.399622 25444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:04.400810 25444 hybrid_clock.cc:648] HybridClock initialized: now 1790433604400783 us; error 42 us; skew 500 ppm
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433604, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:40:04.403455 25444 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:40:04.404341 25444 webserver.cc:460] Webserver started at http://127.16.90.254:42831/ using document root <none> and password file <none>
I20260926 14:40:04.404870 25444 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:04.404949 25444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:04.408798 25444 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:40:04.410843 25459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:04.412082 25444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:40:04.412215 25444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "9afb97a5fda64f24aecd04d8392b0954"
format_stamp: "Formatted at 2026-09-26 14:40:03 on dist-test-slave-x4sn"
server_key: "c2ab4970a80ebe3a69486d8f6075179f"
server_key_iv: "a8b5124100b92d262e62c3b21c9158a8"
server_key_version: "encryptionkey@0"
I20260926 14:40:04.412730 25444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:04.435823 25444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:04.438869 25444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:04.439055 25444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:04.445884 25444 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:34755
I20260926 14:40:04.445919 25511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:34755 every 8 connection(s)
I20260926 14:40:04.446897 25444 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:40:04.447556 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 25444
I20260926 14:40:04.448042 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:43247
--local_ip_for_outbound_sockets=127.16.90.193
--tserver_master_addrs=127.16.90.254:34755
--webserver_port=41277
--webserver_interface=127.16.90.193
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.90.193
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:40:04.451618 25512 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:40:04.453202 25512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: Bootstrap starting.
I20260926 14:40:04.469501 25512 log.cc:824] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: Log is configured to *not* fsync() on all Append() calls
I20260926 14:40:04.476059 25512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:40:04.476400 25512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: Bootstrap complete.
I20260926 14:40:04.480504 25512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afb97a5fda64f24aecd04d8392b0954" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 34755 } }
I20260926 14:40:04.480895 25512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afb97a5fda64f24aecd04d8392b0954, State: Initialized, Role: FOLLOWER
I20260926 14:40:04.481348 25512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afb97a5fda64f24aecd04d8392b0954" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 34755 } }
I20260926 14:40:04.481458 25512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:40:04.481518 25512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:40:04.481611 25512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:40:04.483470 25512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afb97a5fda64f24aecd04d8392b0954" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 34755 } }
I20260926 14:40:04.483800 25512 leader_election.cc:302] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9afb97a5fda64f24aecd04d8392b0954; no voters:
I20260926 14:40:04.484089 25512 leader_election.cc:288] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:40:04.484215 25516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:40:04.484411 25516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [term 2 LEADER]: Becoming Leader. State: Replica: 9afb97a5fda64f24aecd04d8392b0954, State: Running, Role: LEADER
I20260926 14:40:04.484697 25516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afb97a5fda64f24aecd04d8392b0954" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 34755 } }
I20260926 14:40:04.485280 25512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:40:04.485575 25518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9afb97a5fda64f24aecd04d8392b0954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afb97a5fda64f24aecd04d8392b0954" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 34755 } } }
I20260926 14:40:04.485590 25516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9afb97a5fda64f24aecd04d8392b0954. Latest consensus state: current_term: 2 leader_uuid: "9afb97a5fda64f24aecd04d8392b0954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afb97a5fda64f24aecd04d8392b0954" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 34755 } } }
I20260926 14:40:04.485827 25516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [sys.catalog]: This master's current role is: LEADER
I20260926 14:40:04.485889 25518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954 [sys.catalog]: This master's current role is: LEADER
I20260926 14:40:04.489741 25532 catalog_manager.cc:1259] Loaded cluster ID: 204d37aac9734ab39b8334cf0d1ca700
I20260926 14:40:04.489795 25532 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: loading cluster ID for follower catalog manager: success
I20260926 14:40:04.491468 25532 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: acquiring CA information for follower catalog manager: success
I20260926 14:40:04.492640 25532 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:40:04.492743 25520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:40:04.493415 25520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:40:04.493791 25520 catalog_manager.cc:1259] Loaded cluster ID: 204d37aac9734ab39b8334cf0d1ca700
I20260926 14:40:04.493839 25520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:40:04.494359 25520 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:40:04.494724 25520 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9afb97a5fda64f24aecd04d8392b0954: Loaded TSK: 0
I20260926 14:40:04.495309 25520 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:40:04.564312 25514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:04.564595 25514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:04.564657 25514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:04.568706 25514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:04.568773 25514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:04.568854 25514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:40:04.574141 25514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.193.keytab
--principal=oryx/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:43247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=41277
--tserver_master_addrs=127.16.90.254:34755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:04.575163 25514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:04.576174 25514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:04.585377 25538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:04.585404 25539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:04.585376 25541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:04.585804 25514 server_base.cc:1034] running on GCE node
I20260926 14:40:04.586345 25514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:04.587065 25514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:04.588238 25514 hybrid_clock.cc:648] HybridClock initialized: now 1790433604588210 us; error 46 us; skew 500 ppm
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433604, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:40:04.590844 25514 init.cc:377] Logged in from keytab as oryx/127.16.90.193@KRBTEST.COM (short username oryx)
I20260926 14:40:04.591754 25514 webserver.cc:460] Webserver started at http://127.16.90.193:41277/ using document root <none> and password file <none>
I20260926 14:40:04.592295 25514 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:04.592368 25514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:04.596407 25514 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:40:04.598575 25548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:04.599866 25514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:40:04.599968 25514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "b7b417fa63454a01871668aa4738613e"
format_stamp: "Formatted at 2026-09-26 14:40:03 on dist-test-slave-x4sn"
server_key: "c59afcc492881e17d2b24b5d51da4323"
server_key_iv: "15010da740dc3e0d9836a9fa47c2b6a7"
server_key_version: "encryptionkey@0"
I20260926 14:40:04.600526 25514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:04.629066 25514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:04.632290 25514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:04.632462 25514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:04.633059 25514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:04.633877 25514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:04.633924 25514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:04.633965 25514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:04.633993 25514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:04.642603 25514 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:43247
I20260926 14:40:04.642630 25662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:43247 every 8 connection(s)
I20260926 14:40:04.643577 25514 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:40:04.644737 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 25514
I20260926 14:40:04.645143 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:40167
--local_ip_for_outbound_sockets=127.16.90.194
--tserver_master_addrs=127.16.90.254:34755
--webserver_port=42925
--webserver_interface=127.16.90.194
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.90.194
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:40:04.656380 25667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:04.646659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:58573 (local address 127.16.90.254:34755)
0926 14:40:04.647203 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:04.647222 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:04.647269 (+ 47us) server_negotiation.cc:407] Connection header received
0926 14:40:04.647880 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:04.647892 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:04.648190 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:04.648496 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:04.649629 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.650587 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.651708 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.652171 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.652318 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.655622 (+ 3304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:58573: BlockingRecv error: recv got EOF from 127.16.90.193:58573 (error 108)
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":103,"threads_started":1}
I20260926 14:40:04.656388 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:04.645760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:58573)
0926 14:40:04.646643 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:04.646707 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:04.647494 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:04.648584 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:04.648592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:04.648995 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:04.649458 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.649471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.650713 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.650716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.651572 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.651581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.652318 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.652322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.652509 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.653017 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:04.653033 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:04.655388 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":530,"threads_started":1}
W20260926 14:40:04.657012 25663 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:34755 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.90.254:34755: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:04.663435 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:04.657429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:54197)
0926 14:40:04.657586 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:04.657599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:04.657661 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:04.657856 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:04.657858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:04.657972 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:04.658056 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.658060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.659068 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.659090 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.659614 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.659622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.660219 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.660222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.660295 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.660745 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:04.660756 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:04.663340 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260926 14:40:04.663602 25667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:04.657538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:54197 (local address 127.16.90.254:34755)
0926 14:40:04.657650 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:04.657654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:04.657667 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:40:04.657725 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:04.657728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:04.657770 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:04.657850 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:04.658166 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.658982 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.659763 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.660087 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.660206 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.663491 (+ 3285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:54197: BlockingRecv error: recv got EOF from 127.16.90.193:54197 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:04.668753 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:04.663996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:47045)
0926 14:40:04.664196 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:04.664207 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:04.664274 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:04.664467 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:04.664469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:04.664581 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:04.664648 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.664663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.665549 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.665551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.666061 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.666067 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.666546 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.666548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.666602 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.667011 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:04.667025 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:04.668660 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142}
W20260926 14:40:04.668902 25667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:04.664088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:47045 (local address 127.16.90.254:34755)
0926 14:40:04.664196 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:04.664200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:04.664245 (+ 45us) server_negotiation.cc:407] Connection header received
0926 14:40:04.664332 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:04.664334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:04.664374 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:04.664448 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:04.664746 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.665460 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.666157 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.666454 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.666549 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.668799 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:47045: BlockingRecv error: recv got EOF from 127.16.90.193:47045 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 14:40:04.669085 25663 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:40:04.755611 25665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:04.755887 25665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:04.755930 25665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:04.759829 25665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:04.759886 25665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:04.759953 25665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:40:04.765106 25665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.194.keytab
--principal=oryx/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:40167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=42925
--tserver_master_addrs=127.16.90.254:34755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:04.766135 25665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:04.767161 25665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:04.776050 25673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:04.776057 25672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:04.776180 25665 server_base.cc:1034] running on GCE node
W20260926 14:40:04.776057 25675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:04.776619 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:04.777207 25665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:04.778379 25665 hybrid_clock.cc:648] HybridClock initialized: now 1790433604778366 us; error 27 us; skew 500 ppm
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433604, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:40:04.781038 25665 init.cc:377] Logged in from keytab as oryx/127.16.90.194@KRBTEST.COM (short username oryx)
I20260926 14:40:04.781881 25665 webserver.cc:460] Webserver started at http://127.16.90.194:42925/ using document root <none> and password file <none>
I20260926 14:40:04.782421 25665 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:04.782501 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:04.786250 25665 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:40:04.788269 25682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:04.789377 25665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:40:04.789479 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "186550b7fad84412a6ff02d8d9bcdd5a"
format_stamp: "Formatted at 2026-09-26 14:40:03 on dist-test-slave-x4sn"
server_key: "5a81572bc1430cfc990dbfe69f8af99a"
server_key_iv: "c020118c3cb8f561e9f9777ea8a35a3d"
server_key_version: "encryptionkey@0"
I20260926 14:40:04.789978 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:04.823483 25665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:04.826475 25665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:04.826660 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:04.827265 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:04.828116 25665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:04.828186 25665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:04.828240 25665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:04.828269 25665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:04.836534 25665 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:40167
I20260926 14:40:04.836587 25795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:40167 every 8 connection(s)
I20260926 14:40:04.837528 25665 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:40:04.841774 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 25665
I20260926 14:40:04.842250 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:41049
--local_ip_for_outbound_sockets=127.16.90.195
--tserver_master_addrs=127.16.90.254:34755
--webserver_port=33115
--webserver_interface=127.16.90.195
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.90.195
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:40:04.848990 25667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:04.839210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:46823 (local address 127.16.90.254:34755)
0926 14:40:04.839326 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:04.839330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:04.840391 (+ 1061us) server_negotiation.cc:407] Connection header received
0926 14:40:04.841238 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:04.841242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:04.841326 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:04.841408 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:04.842852 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.843591 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.845108 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.845423 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.845523 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.848851 (+ 3328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:46823: BlockingRecv error: recv got EOF from 127.16.90.194:46823 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:40:04.849491 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:04.839561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:46823)
0926 14:40:04.840243 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:04.840282 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:04.841043 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:04.841620 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:04.841630 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:04.842160 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:04.842708 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.842722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.843734 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.843737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.844981 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.844991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.845560 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.845563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.845703 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.846311 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:04.846327 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:04.848694 (+ 2367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":112,"threads_started":1}
W20260926 14:40:04.849965 25796 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:34755 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.90.254:34755: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:04.855296 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:04.850387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:33269)
0926 14:40:04.850542 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:04.850554 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:04.850619 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:04.850833 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:04.850834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:04.850952 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:04.851035 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.851039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.851984 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.851987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.852516 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.852524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.853056 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.853058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.853119 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.853550 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:04.853560 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:04.855209 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260926 14:40:04.855638 25667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:04.850507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:33269 (local address 127.16.90.254:34755)
0926 14:40:04.850630 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:04.850634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:04.850650 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:40:04.850692 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:04.850695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:04.850739 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:04.850826 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:04.851132 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.851888 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.852622 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.852958 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.853064 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.855348 (+ 2284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:33269: BlockingRecv error: recv got EOF from 127.16.90.194:33269 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:04.860558 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:04.855770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:47675)
0926 14:40:04.855865 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:04.855879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:04.855945 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:04.856250 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:04.856253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:04.856408 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:04.856490 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.856495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.857111 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.857114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.857956 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:04.857964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.858362 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.858365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:04.858444 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.858825 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:04.858838 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:04.860449 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 14:40:04.860751 25667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:04.855971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:47675 (local address 127.16.90.254:34755)
0926 14:40:04.856064 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:04.856067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:04.856077 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:40:04.856105 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:04.856108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:04.856147 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:04.856230 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:04.856574 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.857031 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.858071 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:04.858275 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:04.858348 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:04.860569 (+ 2221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:47675: BlockingRecv error: recv got EOF from 127.16.90.194:47675 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 14:40:04.860743 25796 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:40:04.953562 25799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:04.953790 25799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:04.953827 25799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:04.957695 25799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:04.957743 25799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:04.957796 25799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:40:04.962812 25799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/krb5kdc/oryx_127.16.90.195.keytab
--principal=oryx/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:41049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=33115
--tserver_master_addrs=127.16.90.254:34755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:04.963788 25799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:04.964706 25799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:04.973585 25804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:04.973639 25799 server_base.cc:1034] running on GCE node
W20260926 14:40:04.973574 25807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:04.973555 25805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:04.974133 25799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:04.974736 25799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:04.975896 25799 hybrid_clock.cc:648] HybridClock initialized: now 1790433604975880 us; error 27 us; skew 500 ppm
Sep 26 14:40:04 dist-test-slave-x4sn krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433604, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:40:04.978392 25799 init.cc:377] Logged in from keytab as oryx/127.16.90.195@KRBTEST.COM (short username oryx)
I20260926 14:40:04.979207 25799 webserver.cc:460] Webserver started at http://127.16.90.195:33115/ using document root <none> and password file <none>
I20260926 14:40:04.979754 25799 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:04.979836 25799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:04.983470 25799 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:40:04.985424 25814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:04.986653 25799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:40:04.986757 25799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "2cec0cc04daf49eaab7de5551b2bf010"
format_stamp: "Formatted at 2026-09-26 14:40:04 on dist-test-slave-x4sn"
server_key: "2ca98d8b68cf2aea53f68d19e116373e"
server_key_iv: "b02cd4c813d7b04b39249ed1e6616171"
server_key_version: "encryptionkey@0"
I20260926 14:40:04.987283 25799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:05.009147 25799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:05.012075 25799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:05.012235 25799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:05.012779 25799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:05.013553 25799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:05.013598 25799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:05.013636 25799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:05.013660 25799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:05.022161 25799 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:41049
I20260926 14:40:05.022190 25928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:41049 every 8 connection(s)
I20260926 14:40:05.023121 25799 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:40:05.028604 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 25799
Sep 26 14:40:05 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:05 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:40:05.033743 25667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:05.024822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:36745 (local address 127.16.90.254:34755)
0926 14:40:05.024941 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:05.024945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:05.025965 (+ 1020us) server_negotiation.cc:407] Connection header received
0926 14:40:05.026762 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:05.026765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:05.026810 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:05.026879 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:05.028030 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.028521 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.029892 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.030104 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.030189 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:05.033623 (+ 3434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:36745: BlockingRecv error: recv got EOF from 127.16.90.195:36745 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:40:05.034351 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:05.025195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:36745)
0926 14:40:05.025820 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:05.025859 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:05.026598 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:05.027032 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:05.027039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:05.027393 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:05.027909 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:05.027923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.028632 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.028636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:05.029769 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:05.029779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.030239 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.030243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:05.030393 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:05.030929 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:05.030949 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:05.033508 (+ 2559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":106,"threads_started":1}
W20260926 14:40:05.034778 25929 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:34755 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.90.254:34755: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Sep 26 14:40:05 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:05.039188 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:05.029599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.16.90.254:34755)
0926 14:40:05.029831 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:05.029834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:05.029845 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:40:05.029875 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:05.029877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:05.029907 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:05.029983 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:05.030697 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.031420 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.032465 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.032687 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.032781 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:05.034731 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:05.034738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:05.034755 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:40:05.034785 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:05.037176 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:05.037575 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:05.037581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:05.037587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:05.037754 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:05.038030 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:05.038033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:05.038034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:05.038363 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:05.038514 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:05.038829 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 14:40:05.039067 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
Sep 26 14:40:05 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:05.039829 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:05.035186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:54097)
0926 14:40:05.035286 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:05.035298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:05.035367 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:05.035590 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:05.035592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:05.035752 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:05.035834 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:05.035839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.036503 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.036505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:05.037016 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:05.037023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.037409 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.037412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:05.037494 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:05.037898 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:05.037912 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:05.039743 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 14:40:05.039950 25667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:05.035293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:54097 (local address 127.16.90.254:34755)
0926 14:40:05.035408 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:05.035412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:05.035424 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:40:05.035453 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:05.035456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:05.035490 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:05.035562 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:05.035936 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.036418 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.037117 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.037318 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.037402 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:05.039854 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:54097: BlockingRecv error: recv got EOF from 127.16.90.195:54097 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 14:40:05 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:05 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:05.044890 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:05.040453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:43435)
0926 14:40:05.040559 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:05.040573 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:05.040638 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:05.040880 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:05.040883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:05.041070 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:05.041169 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:05.041177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.041983 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.041987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:05.042555 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:05.042565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.042963 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.042966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:05.043024 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:05.043369 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:05.043379 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:05.044803 (+ 1424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:40:05.044998 25667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:05.040519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:43435 (local address 127.16.90.254:34755)
0926 14:40:05.040643 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:05.040648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:05.040670 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:40:05.040701 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:05.040703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:05.040739 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:05.040814 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:05.041284 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.041870 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.042664 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.042859 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.042932 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:05.044906 (+ 1974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:43435: BlockingRecv error: recv got EOF from 127.16.90.195:43435 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 14:40:05.045070 25929 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 14:40:05 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:05 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:05.674677 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:05.669642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:40331)
0926 14:40:05.669893 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:05.669907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:05.669991 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:05.670269 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:05.670271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:05.670437 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:05.670539 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:05.670545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.671266 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.671269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:05.671884 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:05.671893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.672319 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.672321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:05.672385 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:05.672839 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:05.672850 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:05.674558 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
W20260926 14:40:05.674820 25935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:05.669719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:40331 (local address 127.16.90.254:34755)
0926 14:40:05.669965 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:05.669968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:05.669981 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:40:05.670073 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:05.670075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:05.670118 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:05.670204 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:05.670633 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.671156 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.671984 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.672216 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.672295 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:05.674701 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:40331: BlockingRecv error: recv got EOF from 127.16.90.193:40331 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
Sep 26 14:40:05 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:05 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:05.866808 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:05.861341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:40169)
0926 14:40:05.861620 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:05.861636 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:05.861764 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:05.862042 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:05.862046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:05.862226 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:05.862319 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:05.862324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.863054 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.863057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:05.863637 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:05.863645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.864142 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.864146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:05.864234 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:05.864795 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:05.864810 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:05.866653 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
W20260926 14:40:05.866904 25935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:05.861459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:40169 (local address 127.16.90.254:34755)
0926 14:40:05.861622 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:05.861626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:05.861705 (+ 79us) server_negotiation.cc:407] Connection header received
0926 14:40:05.861853 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:05.861856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:05.861902 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:05.861968 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:05.862429 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.862949 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.863774 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:05.864021 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:05.864130 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:05.866804 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:40169: BlockingRecv error: recv got EOF from 127.16.90.194:40169 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 14:40:06 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:06 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:06.051478 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:06.045713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:50339)
0926 14:40:06.046064 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:06.046079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:06.046189 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:06.046409 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:06.046411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:06.046581 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:06.046685 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:06.046689 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:06.047371 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:06.047373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:06.047994 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:06.048002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:06.048424 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:06.048427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:06.048487 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:06.049141 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:06.049163 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:06.051319 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":157,"threads_started":1}
W20260926 14:40:06.051573 25935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:06.045820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:50339 (local address 127.16.90.254:34755)
0926 14:40:06.045951 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:06.045955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:06.046150 (+ 195us) server_negotiation.cc:407] Connection header received
0926 14:40:06.046271 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:06.046274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:06.046319 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:06.046378 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:06.046773 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:06.047284 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:06.048097 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:06.048328 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:06.048409 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:06.051470 (+ 3061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:50339: BlockingRecv error: recv got EOF from 127.16.90.195:50339 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 14:40:06 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:06 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:06.681186 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:06.675506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:35207)
0926 14:40:06.675836 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:06.675852 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:06.675951 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:06.676260 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:06.676262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:06.676446 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:06.676566 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:06.676572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:06.677344 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:06.677347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:06.677925 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:06.677934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:06.678398 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:06.678402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:06.678473 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:06.679050 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:06.679061 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:06.681021 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":86,"threads_started":1}
W20260926 14:40:06.681288 25939 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:06.675626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:35207 (local address 127.16.90.254:34755)
0926 14:40:06.675889 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:06.675892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:06.675906 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:40:06.676083 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:06.676086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:06.676134 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:06.676216 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:06.676682 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:06.677234 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:06.678050 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:06.678292 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:06.678372 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:06.681170 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:35207: BlockingRecv error: recv got EOF from 127.16.90.193:35207 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":108,"threads_started":1}
Sep 26 14:40:06 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:06 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:06.872810 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:06.867589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:57891)
0926 14:40:06.867896 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:06.867909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:06.868009 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:06.868262 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:06.868265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:06.868434 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:06.868533 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:06.868537 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:06.869258 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:06.869261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:06.869831 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:06.869840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:06.870252 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:06.870255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:06.870323 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:06.870791 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:06.870803 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:06.872626 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":124,"threads_started":1}
W20260926 14:40:06.872880 25939 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:06.867691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:57891 (local address 127.16.90.254:34755)
0926 14:40:06.867814 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:06.867819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:06.867991 (+ 172us) server_negotiation.cc:407] Connection header received
0926 14:40:06.868119 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:06.868122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:06.868169 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:06.868240 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:06.868612 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:06.869136 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:06.869932 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:06.870147 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:06.870227 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:06.872781 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:57891: BlockingRecv error: recv got EOF from 127.16.90.194:57891 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 14:40:07 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:07 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:07.057839 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:07.052328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:41367)
0926 14:40:07.052648 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:07.052748 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:07.052864 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:07.053077 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:07.053079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:07.053291 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:07.053387 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:07.053393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:07.054186 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:07.054189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:07.054766 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:07.054774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:07.055277 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:07.055280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:07.055343 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:07.055898 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:07.055910 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:07.057658 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":111,"threads_started":1}
W20260926 14:40:07.057904 25939 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:07.052518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:41367 (local address 127.16.90.254:34755)
0926 14:40:07.052663 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:07.052667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:07.052825 (+ 158us) server_negotiation.cc:407] Connection header received
0926 14:40:07.052937 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:07.052939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:07.052983 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:07.053055 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:07.053514 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:07.054081 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:07.054893 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:07.055158 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:07.055247 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:07.057802 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:41367: BlockingRecv error: recv got EOF from 127.16.90.195:41367 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 14:40:07 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:07 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:07.687290 25942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:07.681914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:35961)
0926 14:40:07.682209 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:07.682223 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:07.682315 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:07.682564 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:07.682566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:07.682735 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:07.682830 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:07.682836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:07.683626 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:07.683628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:07.684224 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:07.684233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:07.684636 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:07.684639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:07.684698 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:07.685159 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:07.685172 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:07.687112 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":117,"threads_started":1}
W20260926 14:40:07.687387 25943 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:07.681989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:35961 (local address 127.16.90.254:34755)
0926 14:40:07.682252 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:07.682255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:07.682272 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:40:07.682393 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:07.682396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:07.682440 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:07.682544 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:07.682951 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:07.683519 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:07.684323 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:07.684552 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:07.684623 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:07.687259 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:35961: BlockingRecv error: recv got EOF from 127.16.90.193:35961 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
Sep 26 14:40:07 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:07 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:07.878789 25944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:07.873615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:47549)
0926 14:40:07.873866 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:07.873882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:07.873989 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:07.874338 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:07.874341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:07.874579 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:07.874700 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:07.874706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:07.875441 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:07.875443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:07.876038 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:07.876046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:07.876484 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:07.876487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:07.876557 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:07.877005 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:07.877016 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:07.878670 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":65,"threads_started":1}
W20260926 14:40:07.878887 25943 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:07.873751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:47549 (local address 127.16.90.254:34755)
0926 14:40:07.873904 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:07.873907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:07.873933 (+ 26us) server_negotiation.cc:407] Connection header received
0926 14:40:07.874148 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:07.874150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:07.874200 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:07.874276 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:07.874802 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:07.875347 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:07.876138 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:07.876366 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:07.876446 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:07.878794 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:47549: BlockingRecv error: recv got EOF from 127.16.90.194:47549 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 14:40:08 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:08 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:08.063644 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:08.058599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:34755)
0926 14:40:08.058841 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:08.058855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:08.058954 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:08.059155 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:08.059157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:08.059334 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:08.059431 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:08.059437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:08.060188 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:08.060190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:08.060733 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:08.060741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:08.061148 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:08.061151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:08.061213 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:08.061652 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:08.061663 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:08.063525 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
W20260926 14:40:08.063832 25943 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:08.058709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:34755 (local address 127.16.90.254:34755)
0926 14:40:08.058847 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:08.058851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:08.058907 (+ 56us) server_negotiation.cc:407] Connection header received
0926 14:40:08.059016 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:08.059018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:08.059060 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:08.059122 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:08.059528 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:08.060095 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:08.060835 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:08.061045 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:08.061125 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:08.063707 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:34755: BlockingRecv error: recv got EOF from 127.16.90.195:34755 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 14:40:08 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:08 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:08.700280 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:08.688037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:56509)
0926 14:40:08.688317 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:08.688332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:08.688448 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:08.688716 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:08.688719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:08.688927 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:08.689048 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:08.689057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:08.689778 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:08.689780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:08.690350 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:08.690358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:08.690853 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:08.690856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:08.690915 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:08.698254 (+ 7339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:08.698266 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:08.700132 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":100,"threads_started":1}
W20260926 14:40:08.700390 25947 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:08.688086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:56509 (local address 127.16.90.254:34755)
0926 14:40:08.688315 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:08.688320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:08.688400 (+ 80us) server_negotiation.cc:407] Connection header received
0926 14:40:08.688535 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:08.688538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:08.688593 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:08.688673 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:08.689161 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:08.689695 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:08.690446 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:08.690766 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:08.690844 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:08.700260 (+ 9416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:56509: BlockingRecv error: recv got EOF from 127.16.90.193:56509 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":53,"threads_started":1}
Sep 26 14:40:08 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:08 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:08.885196 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:08.879598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:42111)
0926 14:40:08.879933 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:08.879948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:08.880061 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:08.880336 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:08.880339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:08.880533 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:08.880628 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:08.880634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:08.881395 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:08.881399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:08.882160 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:08.882172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:08.882706 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:08.882709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:08.882775 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:08.883236 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:08.883249 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:08.885065 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":157,"threads_started":1}
W20260926 14:40:08.885371 25947 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:08.879729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:42111 (local address 127.16.90.254:34755)
0926 14:40:08.879859 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:08.879863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:08.880019 (+ 156us) server_negotiation.cc:407] Connection header received
0926 14:40:08.880180 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:08.880182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:08.880229 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:08.880295 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:08.880749 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:08.881281 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:08.882272 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:08.882586 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:08.882712 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:08.885247 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:42111: BlockingRecv error: recv got EOF from 127.16.90.194:42111 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 14:40:09 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:09 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:09.070044 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:09.064625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:47997)
0926 14:40:09.064877 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:09.064891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:09.064999 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:09.065291 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:09.065294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:09.065510 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:09.065613 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:09.065619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:09.066474 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:09.066477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:09.067070 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:09.067080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:09.067494 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:09.067497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:09.067557 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:09.068080 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:09.068092 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:09.069927 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
W20260926 14:40:09.070189 25947 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:09.064744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:47997 (local address 127.16.90.254:34755)
0926 14:40:09.064911 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:09.064914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:09.064948 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:40:09.065103 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:09.065107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:09.065164 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:09.065261 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:09.065715 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:09.066337 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:09.067179 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:09.067401 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:09.067485 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:09.070061 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:47997: BlockingRecv error: recv got EOF from 127.16.90.195:47997 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 14:40:09 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:09 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:09.706337 25950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:09.701038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:58181)
0926 14:40:09.701389 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:09.701403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:09.701515 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:09.701882 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:09.701885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:09.702064 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:09.702163 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:09.702169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:09.702909 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:09.702911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:09.703536 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:09.703546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:09.704000 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:09.704003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:09.704060 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:09.704586 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:09.704599 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:09.706222 (+ 1623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":100,"threads_started":1}
W20260926 14:40:09.706455 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:09.701074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:58181 (local address 127.16.90.254:34755)
0926 14:40:09.701350 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:09.701354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:09.701499 (+ 145us) server_negotiation.cc:407] Connection header received
0926 14:40:09.701606 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:09.701608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:09.701653 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:09.701760 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:09.702281 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:09.702828 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:09.703633 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:09.703912 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:09.703991 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:09.706344 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:58181: BlockingRecv error: recv got EOF from 127.16.90.193:58181 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
Sep 26 14:40:09 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:09 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:09.890868 25952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:09.886020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:59483)
0926 14:40:09.886295 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:09.886308 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:09.886397 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:09.886593 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:09.886595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:09.886770 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:09.886860 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:09.886865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:09.887612 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:09.887614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:09.888217 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:09.888226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:09.888583 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:09.888586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:09.888643 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:09.889069 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:09.889080 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:09.890754 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
W20260926 14:40:09.890975 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:09.886131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:59483 (local address 127.16.90.254:34755)
0926 14:40:09.886292 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:09.886295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:09.886357 (+ 62us) server_negotiation.cc:407] Connection header received
0926 14:40:09.886472 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:09.886474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:09.886514 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:09.886570 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:09.886954 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:09.887513 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:09.888302 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:09.888507 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:09.888577 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:09.890877 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:59483: BlockingRecv error: recv got EOF from 127.16.90.194:59483 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 14:40:10 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:10 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:10.075984 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:10.070788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:54799)
0926 14:40:10.071051 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:10.071066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:10.071189 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:10.071430 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:10.071433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:10.071631 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:10.071760 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:10.071766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:10.072501 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:10.072505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:10.073060 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:10.073068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:10.073490 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:10.073494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:10.073575 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:10.074054 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:10.074068 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:10.075867 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
W20260926 14:40:10.076222 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:10.070884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:54799 (local address 127.16.90.254:34755)
0926 14:40:10.071034 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:10.071038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:10.071139 (+ 101us) server_negotiation.cc:407] Connection header received
0926 14:40:10.071278 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:10.071281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:10.071329 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:10.071396 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:10.071873 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:10.072397 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:10.073174 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:10.073394 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:10.073479 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:10.076008 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:54799: BlockingRecv error: recv got EOF from 127.16.90.195:54799 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 14:40:10 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:10 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:10.711843 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:10.707028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:44653)
0926 14:40:10.707280 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:10.707295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:10.707385 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:10.707622 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:10.707624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:10.707835 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:10.707933 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:10.707939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:10.708665 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:10.708667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:10.709208 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:10.709215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:10.709578 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:10.709580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:10.709635 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:10.710133 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:10.710145 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:10.711726 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":66,"threads_started":1}
W20260926 14:40:10.712016 25955 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:10.707092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:44653 (local address 127.16.90.254:34755)
0926 14:40:10.707331 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:10.707335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:10.707347 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:40:10.707469 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:10.707471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:10.707523 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:10.707599 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:10.708031 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:10.708579 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:10.709289 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:10.709497 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:10.709571 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:10.711877 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:44653: BlockingRecv error: recv got EOF from 127.16.90.193:44653 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
Sep 26 14:40:10 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:10 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:10.896899 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:10.891722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:52909)
0926 14:40:10.891983 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:10.891997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:10.892114 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:10.892388 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:10.892391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:10.892601 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:10.892702 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:10.892708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:10.893499 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:10.893502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:10.894112 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:10.894123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:10.894529 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:10.894531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:10.894591 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:10.895076 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:10.895088 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:10.896779 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":69,"threads_started":1}
W20260926 14:40:10.897048 25955 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:10.891811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:52909 (local address 127.16.90.254:34755)
0926 14:40:10.891972 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:10.891975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:10.892062 (+ 87us) server_negotiation.cc:407] Connection header received
0926 14:40:10.892237 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:10.892239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:10.892289 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:10.892364 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:10.892794 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:10.893375 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:10.894229 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:10.894441 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:10.894527 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:10.896906 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:52909: BlockingRecv error: recv got EOF from 127.16.90.194:52909 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 14:40:11 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:11 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:11.083477 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:11.076793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:45109)
0926 14:40:11.077217 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:11.077228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:11.077369 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:11.077924 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:11.077928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:11.078230 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:11.078362 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:11.078370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:11.079129 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:11.079132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:11.080073 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:11.080085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:11.080514 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:11.080518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:11.080605 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:11.081301 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:11.081313 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:11.083333 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":149,"threads_started":1}
W20260926 14:40:11.083591 25955 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:11.076881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:45109 (local address 127.16.90.254:34755)
0926 14:40:11.077054 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:11.077058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:11.077644 (+ 586us) server_negotiation.cc:407] Connection header received
0926 14:40:11.077714 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:11.077717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:11.077775 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:11.077857 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:11.078490 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:11.079019 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:11.080191 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:11.080410 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:11.080487 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:11.083468 (+ 2981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:45109: BlockingRecv error: recv got EOF from 127.16.90.195:45109 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 14:40:11 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:11 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:11.718180 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:11.712690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:40601)
0926 14:40:11.712979 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:11.712993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:11.713114 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:11.713391 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:11.713394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:11.713584 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:11.713688 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:11.713695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:11.714472 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:11.714475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:11.715024 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:11.715032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:11.715454 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:11.715458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:11.715526 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:11.716303 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:11.716315 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:11.718066 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":69,"threads_started":1}
W20260926 14:40:11.718319 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:11.712824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:40601 (local address 127.16.90.254:34755)
0926 14:40:11.713054 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:11.713059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:11.713073 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:40:11.713209 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:11.713212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:11.713267 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:11.713357 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:11.713801 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:11.714355 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:11.715120 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:11.715347 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:11.715420 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:11.718187 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:40601: BlockingRecv error: recv got EOF from 127.16.90.193:40601 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
Sep 26 14:40:11 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:11 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:11.902774 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:11.897607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:57657)
0926 14:40:11.897951 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:11.897964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:11.898074 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:11.898316 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:11.898319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:11.898532 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:11.898629 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:11.898635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:11.899375 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:11.899379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:11.899982 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:11.899991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:11.900411 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:11.900415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:11.900499 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:11.900965 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:11.900977 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:11.902660 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":80,"threads_started":1}
W20260926 14:40:11.902904 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:11.897716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:57657 (local address 127.16.90.254:34755)
0926 14:40:11.897876 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:11.897881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:11.898067 (+ 186us) server_negotiation.cc:407] Connection header received
0926 14:40:11.898165 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:11.898168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:11.898214 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:11.898279 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:11.898742 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:11.899272 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:11.900096 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:11.900312 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:11.900398 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:11.902790 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:57657: BlockingRecv error: recv got EOF from 127.16.90.194:57657 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 14:40:12 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:12 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:12.089785 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:12.084320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:44513)
0926 14:40:12.084621 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:12.084633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:12.084732 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:12.084999 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:12.085001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:12.085211 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:12.085308 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:12.085314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:12.086209 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:12.086212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:12.086808 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:12.086818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:12.087260 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:12.087263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:12.087339 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:12.087881 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:12.087893 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:12.089671 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":84,"threads_started":1}
W20260926 14:40:12.089915 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:12.084421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:44513 (local address 127.16.90.254:34755)
0926 14:40:12.084542 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:12.084546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:12.084724 (+ 178us) server_negotiation.cc:407] Connection header received
0926 14:40:12.084835 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:12.084838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:12.084897 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:12.084967 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:12.085418 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:12.086041 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:12.086912 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:12.087170 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:12.087248 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:12.089806 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:44513: BlockingRecv error: recv got EOF from 127.16.90.195:44513 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 14:40:12 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:12 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:12.724364 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:12.718862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:47647)
0926 14:40:12.719197 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:12.719215 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:12.719317 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:12.719533 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:12.719535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:12.719740 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:12.719845 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:12.719851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:12.720567 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:12.720571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:12.721148 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:12.721157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:12.721547 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:12.721550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:12.721611 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:12.722305 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:12.722319 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:12.724244 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":82,"threads_started":1}
W20260926 14:40:12.724502 25963 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:12.719035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:47647 (local address 127.16.90.254:34755)
0926 14:40:12.719322 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:12.719325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:12.719338 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:40:12.719376 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:12.719379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:12.719417 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:12.719490 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:12.719931 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:12.720458 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:12.721247 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:12.721462 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:12.721540 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:12.724370 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:47647: BlockingRecv error: recv got EOF from 127.16.90.193:47647 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":124,"threads_started":1}
Sep 26 14:40:12 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:12 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:12.909210 25964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:12.903554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:41331)
0926 14:40:12.903849 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:12.903863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:12.903976 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:12.904310 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:12.904313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:12.904536 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:12.904650 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:12.904657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:12.905605 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:12.905608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:12.906190 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:12.906199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:12.906582 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:12.906584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:12.906648 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:12.907111 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:12.907123 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:12.909081 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":111,"threads_started":1}
W20260926 14:40:12.909348 25963 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:12.903652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:41331 (local address 127.16.90.254:34755)
0926 14:40:12.903842 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:12.903846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:12.903954 (+ 108us) server_negotiation.cc:407] Connection header received
0926 14:40:12.904106 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:12.904109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:12.904156 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:12.904262 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:12.904764 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:12.905490 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:12.906282 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:12.906489 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:12.906562 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:12.909228 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:41331: BlockingRecv error: recv got EOF from 127.16.90.194:41331 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 14:40:13 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:13 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:13.095721 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:13.090529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:50033)
0926 14:40:13.090793 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:13.090806 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:13.090904 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:13.091110 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:13.091112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:13.091306 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:13.091422 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:13.091428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:13.092203 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:13.092206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:13.092876 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:13.092887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:13.093320 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:13.093323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:13.093385 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:13.093938 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:13.093951 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:13.095559 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":73,"threads_started":1}
W20260926 14:40:13.095821 25963 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:13.090591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:50033 (local address 127.16.90.254:34755)
0926 14:40:13.090740 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:13.090745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:13.090869 (+ 124us) server_negotiation.cc:407] Connection header received
0926 14:40:13.090972 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:13.090974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:13.091020 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:13.091088 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:13.091522 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:13.092107 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:13.092987 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:13.093224 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:13.093299 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:13.095723 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:50033: BlockingRecv error: recv got EOF from 127.16.90.195:50033 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 14:40:13 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:13 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:13.735147 25966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:13.725129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:49759)
0926 14:40:13.725409 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:13.725422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:13.725528 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:13.725877 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:13.725880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:13.726057 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:13.726190 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:13.726196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:13.726914 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:13.726917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:13.727509 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:13.727518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:13.727961 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:13.727963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:13.728024 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:13.732917 (+ 4893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:13.732934 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:13.734995 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
W20260926 14:40:13.735249 25967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:13.725260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:49759 (local address 127.16.90.254:34755)
0926 14:40:13.725483 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:13.725486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:13.725499 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:40:13.725672 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:13.725682 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:13.725743 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:13.725828 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:13.726281 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:13.726818 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:13.727618 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:13.727878 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:13.727957 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:13.735132 (+ 7175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:49759: BlockingRecv error: recv got EOF from 127.16.90.193:49759 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":63,"threads_started":1}
Sep 26 14:40:13 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:13 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:13.915551 25968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:13.909960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:33523)
0926 14:40:13.910300 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:13.910315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:13.910430 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:13.910687 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:13.910690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:13.910889 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:13.910985 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:13.910991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:13.911799 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:13.911803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:13.912367 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:13.912376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:13.912806 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:13.912809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:13.912893 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:13.913391 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:13.913402 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:13.915373 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":149,"threads_started":1}
W20260926 14:40:13.915633 25967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:13.910085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:33523 (local address 127.16.90.254:34755)
0926 14:40:13.910271 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:13.910275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:13.910386 (+ 111us) server_negotiation.cc:407] Connection header received
0926 14:40:13.910517 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:13.910520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:13.910567 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:13.910633 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:13.911102 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:13.911656 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:13.912484 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:13.912706 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:13.912793 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:13.915531 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:33523: BlockingRecv error: recv got EOF from 127.16.90.194:33523 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 14:40:14 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:14 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:14.101871 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:14.096538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:57255)
0926 14:40:14.096792 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:14.096805 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:14.096901 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:14.097231 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:14.097234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:14.097413 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:14.097509 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:14.097515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:14.098254 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:14.098256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:14.098820 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:14.098827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:14.099262 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:14.099265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:14.099335 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:14.099915 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:14.099927 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:14.101734 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
W20260926 14:40:14.101979 25967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:14.096640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:57255 (local address 127.16.90.254:34755)
0926 14:40:14.096762 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:14.096765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:14.096899 (+ 134us) server_negotiation.cc:407] Connection header received
0926 14:40:14.096981 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:14.096983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:14.097034 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:14.097097 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:14.097615 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:14.098125 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:14.098906 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:14.099112 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:14.099210 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:14.101877 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:57255: BlockingRecv error: recv got EOF from 127.16.90.195:57255 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 14:40:14 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:14 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:14.741767 25970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:14.736046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:32851)
0926 14:40:14.736346 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:14.736362 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:14.736465 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:14.736951 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:14.736954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:14.737195 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:14.737319 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:14.737327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:14.738307 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:14.738310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:14.738884 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:14.738893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:14.739284 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:14.739287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:14.739343 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:14.739838 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:14.739849 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:14.741633 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":75,"threads_started":1}
W20260926 14:40:14.741904 25971 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:14.736163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:32851 (local address 127.16.90.254:34755)
0926 14:40:14.736448 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:14.736453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:14.736470 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:40:14.736602 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:14.736605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:14.736691 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:14.736834 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:14.737542 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:14.738153 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:14.738979 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:14.739204 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:14.739272 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:14.741774 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:32851: BlockingRecv error: recv got EOF from 127.16.90.193:32851 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
Sep 26 14:40:14 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:14 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:14.921578 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:14.916428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:56609)
0926 14:40:14.916690 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:14.916703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:14.916803 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:14.917055 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:14.917058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:14.917230 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:14.917328 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:14.917333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:14.918081 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:14.918085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:14.918658 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:14.918667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:14.919029 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:14.919031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:14.919097 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:14.919543 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:14.919554 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:14.921449 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
W20260926 14:40:14.921691 25971 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:14.916511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:56609 (local address 127.16.90.254:34755)
0926 14:40:14.916644 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:14.916649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:14.916806 (+ 157us) server_negotiation.cc:407] Connection header received
0926 14:40:14.916868 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:14.916870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:14.916917 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:14.916988 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:14.917446 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:14.917978 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:14.918747 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:14.918950 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:14.919017 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:14.921592 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:56609: BlockingRecv error: recv got EOF from 127.16.90.194:56609 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 14:40:15 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:15 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:15.108222 25973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:15.102636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:59519)
0926 14:40:15.102912 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:15.102928 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:15.103044 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:15.103292 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:15.103295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:15.103485 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:15.103589 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:15.103595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:15.104399 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:15.104402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:15.104987 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:15.104996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:15.105407 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:15.105411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:15.105487 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:15.106038 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:15.106052 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:15.108063 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
W20260926 14:40:15.108315 25971 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:15.102803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:59519 (local address 127.16.90.254:34755)
0926 14:40:15.102928 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:15.102933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:15.103034 (+ 101us) server_negotiation.cc:407] Connection header received
0926 14:40:15.103138 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:15.103147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:15.103193 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:15.103258 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:15.103728 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:15.104298 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:15.105093 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:15.105312 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:15.105397 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:15.108216 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:59519: BlockingRecv error: recv got EOF from 127.16.90.195:59519 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 14:40:15 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:15 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:15.747915 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:15.742468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:47455)
0926 14:40:15.742735 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:15.742748 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:15.742834 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:15.743064 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:15.743066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:15.743242 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:15.743338 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:15.743344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:15.744185 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:15.744189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:15.744756 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:15.744765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:15.745233 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:15.745236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:15.745296 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:15.745788 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:15.745800 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:15.747761 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
W20260926 14:40:15.748034 25975 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:15.742592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:47455 (local address 127.16.90.254:34755)
0926 14:40:15.742825 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:15.742828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:15.742844 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:40:15.742901 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:15.742903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:15.742946 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:15.743043 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:15.743442 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:15.744047 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:15.744878 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:15.745142 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:15.745226 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:15.747905 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:47455: BlockingRecv error: recv got EOF from 127.16.90.193:47455 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
Sep 26 14:40:15 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:15 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:15.927605 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:15.922427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:44849)
0926 14:40:15.922734 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:15.922747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:15.922855 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:15.923096 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:15.923099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:15.923300 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:15.923398 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:15.923404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:15.924214 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:15.924218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:15.924777 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:15.924785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:15.925209 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:15.925212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:15.925294 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:15.925777 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:15.925788 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:15.927430 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
W20260926 14:40:15.927695 25975 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:15.922519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:44849 (local address 127.16.90.254:34755)
0926 14:40:15.922671 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:15.922676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:15.922816 (+ 140us) server_negotiation.cc:407] Connection header received
0926 14:40:15.922940 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:15.922943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:15.922992 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:15.923058 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:15.923511 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:15.924085 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:15.924890 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:15.925109 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:15.925204 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:15.927570 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:44849: BlockingRecv error: recv got EOF from 127.16.90.194:44849 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 14:40:16 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:16 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:16.113991 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:16.108941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:42823)
0926 14:40:16.109248 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:16.109261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:16.109358 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:16.109588 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:16.109591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:16.109782 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:16.109880 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:16.109886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:16.110557 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:16.110559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:16.111108 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:16.111115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:16.111480 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:16.111482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:16.111537 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:16.112011 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:16.112022 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:16.113859 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
W20260926 14:40:16.114204 25975 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:16.109039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:42823 (local address 127.16.90.254:34755)
0926 14:40:16.109177 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:16.109181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:16.109322 (+ 141us) server_negotiation.cc:407] Connection header received
0926 14:40:16.109428 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:16.109430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:16.109475 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:16.109544 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:16.109968 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:16.110478 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:16.111195 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:16.111402 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:16.111473 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:16.114000 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:42823: BlockingRecv error: recv got EOF from 127.16.90.195:42823 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 14:40:16 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:16 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:16.753582 25979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:16.748714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:47955)
0926 14:40:16.748994 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:16.749008 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:16.749096 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:16.749318 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:16.749320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:16.749490 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:16.749587 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:16.749592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:16.750274 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:16.750276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:16.750831 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:16.750839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:16.751195 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:16.751197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:16.751254 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:16.751732 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:16.751743 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:16.753467 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
W20260926 14:40:16.753731 25980 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:16.748814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:47955 (local address 127.16.90.254:34755)
0926 14:40:16.749039 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:16.749042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:16.749056 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:40:16.749170 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:16.749173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:16.749218 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:16.749289 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:16.749688 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:16.750194 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:16.750914 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:16.751119 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:16.751181 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:16.753602 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:47955: BlockingRecv error: recv got EOF from 127.16.90.193:47955 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":58,"threads_started":1}
Sep 26 14:40:16 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:16 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:16.933601 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:16.928444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:48901)
0926 14:40:16.928743 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:16.928760 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:16.928873 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:16.929108 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:16.929110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:16.929327 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:16.929421 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:16.929427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:16.930164 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:16.930168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:16.930736 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:16.930744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:16.931167 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:16.931171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:16.931273 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:16.931766 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:16.931779 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:16.933479 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
W20260926 14:40:16.933722 25980 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:16.928485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:48901 (local address 127.16.90.254:34755)
0926 14:40:16.928622 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:16.928627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:16.928837 (+ 210us) server_negotiation.cc:407] Connection header received
0926 14:40:16.928961 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:16.928964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:16.929013 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:16.929078 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:16.929536 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:16.930060 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:16.930848 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:16.931070 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:16.931165 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:16.933613 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:48901: BlockingRecv error: recv got EOF from 127.16.90.194:48901 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 14:40:17 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:17 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:17.121399 25982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:17.114667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:38655)
0926 14:40:17.114910 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:17.114923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:17.115022 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:17.115298 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:17.115300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:17.115510 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:17.115640 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:17.115647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:17.116457 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:17.116461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:17.117176 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:17.117187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:17.117832 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:17.117836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:17.117926 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:17.118557 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:17.118572 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:17.121280 (+ 2708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
W20260926 14:40:17.121869 25980 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:17.114741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:38655 (local address 127.16.90.254:34755)
0926 14:40:17.114865 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:17.114868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:17.114982 (+ 114us) server_negotiation.cc:407] Connection header received
0926 14:40:17.115091 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:17.115093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:17.115137 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:17.115232 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:17.115774 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:17.116359 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:17.117287 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:17.117595 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:17.117693 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:17.121752 (+ 4059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:38655: BlockingRecv error: recv got EOF from 127.16.90.195:38655 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 14:40:17 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:17 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:17.760156 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:17.754449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:56027)
0926 14:40:17.754761 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:17.754775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:17.754877 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:17.755141 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:17.755143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:17.755319 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:17.755412 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:17.755417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:17.756277 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:17.756280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:17.756868 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:17.756878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:17.757389 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:17.757392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:17.757458 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:17.758067 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:17.758081 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:17.760026 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
W20260926 14:40:17.760350 25984 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:17.754568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:56027 (local address 127.16.90.254:34755)
0926 14:40:17.754796 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:17.754800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:17.754833 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:40:17.754957 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:17.754959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:17.755012 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:17.755097 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:17.755520 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:17.756169 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:17.756988 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:17.757251 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:17.757348 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:17.760204 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:56027: BlockingRecv error: recv got EOF from 127.16.90.193:56027 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":63,"threads_started":1}
Sep 26 14:40:17 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:17 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:17.939555 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:17.934292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:51159)
0926 14:40:17.934579 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:17.934596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:17.934717 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:17.934957 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:17.934961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:17.935159 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:17.935264 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:17.935270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:17.936080 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:17.936083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:17.936655 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:17.936663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:17.937095 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:17.937099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:17.937184 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:17.937654 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:17.937665 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:17.939405 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
W20260926 14:40:17.939661 25984 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:17.934467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:51159 (local address 127.16.90.254:34755)
0926 14:40:17.934605 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:17.934610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:17.934665 (+ 55us) server_negotiation.cc:407] Connection header received
0926 14:40:17.934806 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:17.934809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:17.934856 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:17.934924 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:17.935380 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:17.935966 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:17.936772 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:17.936993 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:17.937080 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:17.939559 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:51159: BlockingRecv error: recv got EOF from 127.16.90.194:51159 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 14:40:18 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:18 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:18.130743 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:18.122121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:51085)
0926 14:40:18.124489 (+ 2368us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:18.124505 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:18.124615 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:18.124879 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:18.124882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:18.125094 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:18.125212 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:18.125220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:18.126576 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:18.126580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:18.127458 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:18.127470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:18.128033 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:18.128037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:18.128129 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:18.128685 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:18.128701 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:18.130626 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2280,"thread_start_us":72,"threads_started":1}
W20260926 14:40:18.130869 25984 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:18.124461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:51085 (local address 127.16.90.254:34755)
0926 14:40:18.124581 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:18.124585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:18.124597 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:40:18.124707 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:18.124710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:18.124764 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:18.124837 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:18.125339 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:18.126090 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:18.127597 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:18.127929 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:18.128030 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:18.130754 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:51085: BlockingRecv error: recv got EOF from 127.16.90.195:51085 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 14:40:18 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:18 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:18.769097 25987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:18.760892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:51857)
0926 14:40:18.761193 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:18.761208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:18.761296 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:18.761489 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:18.761491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:18.761671 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:18.761777 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:18.761783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:18.762461 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:18.762463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:18.763034 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:18.763041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:18.763403 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:18.763405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:18.763465 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:18.767300 (+ 3835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:18.767312 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:18.768969 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":107,"threads_started":1}
W20260926 14:40:18.769205 25988 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:18.761001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:51857 (local address 127.16.90.254:34755)
0926 14:40:18.761299 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:18.761302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:18.761316 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:40:18.761351 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:18.761353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:18.761392 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:18.761464 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:18.761867 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:18.762384 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:18.763115 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:18.763323 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:18.763389 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:18.769097 (+ 5708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:51857: BlockingRecv error: recv got EOF from 127.16.90.193:51857 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":63,"threads_started":1}
Sep 26 14:40:18 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:18 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:18.945272 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:18.940408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:38499)
0926 14:40:18.940684 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:18.940698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:18.940797 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:18.941003 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:18.941005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:18.941200 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:18.941296 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:18.941301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:18.942001 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:18.942003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:18.942585 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:18.942593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:18.942963 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:18.942966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:18.943022 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:18.943476 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:18.943486 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:18.945141 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
W20260926 14:40:18.945377 25988 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:18.940491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:38499 (local address 127.16.90.254:34755)
0926 14:40:18.940633 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:18.940638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:18.940761 (+ 123us) server_negotiation.cc:407] Connection header received
0926 14:40:18.940866 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:18.940868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:18.940911 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:18.940974 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:18.941383 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:18.941924 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:18.942678 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:18.942888 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:18.942957 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:18.945280 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:38499: BlockingRecv error: recv got EOF from 127.16.90.194:38499 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 14:40:19 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:19 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.195@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:19.136878 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:19.131461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.195:44285)
0926 14:40:19.131795 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:19.131810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:19.131926 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:19.132200 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:19.132203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:19.132420 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:19.132537 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:19.132545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:19.133441 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:19.133444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:19.134025 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:19.134033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:19.134399 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:19.134401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:19.134457 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:19.135018 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:19.135029 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:19.136756 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":97,"threads_started":1}
W20260926 14:40:19.137115 25988 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:19.131564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:44285 (local address 127.16.90.254:34755)
0926 14:40:19.131771 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:19.131776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:19.131883 (+ 107us) server_negotiation.cc:407] Connection header received
0926 14:40:19.132022 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:19.132025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:19.132078 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:19.132139 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:19.132657 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:19.133338 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:19.134113 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:19.134324 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:19.134392 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:19.136891 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.195:44285: BlockingRecv error: recv got EOF from 127.16.90.195:44285 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 26 14:40:19 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:19 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.193@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:19.774873 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:19.769966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.193:49491)
0926 14:40:19.770238 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:19.770251 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:19.770345 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:19.770551 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:19.770553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:19.770735 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:19.770852 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:19.770859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:19.771587 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:19.771589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:19.772219 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:19.772228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:19.772664 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:19.772667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:19.772724 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:19.773167 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:19.773179 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:19.774760 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
W20260926 14:40:19.775008 25992 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:19.770024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:49491 (local address 127.16.90.254:34755)
0926 14:40:19.770238 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:19.770241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:19.770310 (+ 69us) server_negotiation.cc:407] Connection header received
0926 14:40:19.770411 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:19.770413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:19.770452 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:19.770525 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:19.770963 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:19.771499 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:19.772328 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:19.772562 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:19.772650 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:19.774892 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.193:49491: BlockingRecv error: recv got EOF from 127.16.90.193:49491 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":61,"threads_started":1}
Sep 26 14:40:19 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:40:19 dist-test-slave-x4sn krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.90.194@KRBTEST.COM for oryx/127.16.90.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:40:19.950898 25993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:19.946088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:34755 (local address 127.16.90.194:51521)
0926 14:40:19.946349 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:19.946364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:19.946470 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:19.946713 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:19.946716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:19.946877 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:19.946972 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:19.946977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:19.947661 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:19.947694 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:19.948245 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:19.948254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:19.948639 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:19.948641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:19.948708 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:19.949161 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:19.949172 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:19.950785 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.90.254:34755: Server oryx/127.16.90.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
W20260926 14:40:19.951099 25992 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:19.946134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:51521 (local address 127.16.90.254:34755)
0926 14:40:19.946282 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:19.946288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:19.946421 (+ 133us) server_negotiation.cc:407] Connection header received
0926 14:40:19.946557 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:19.946560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:19.946608 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:19.946676 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:19.947055 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:19.947581 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:19.948334 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:19.948556 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:19.948640 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:19.950927 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.90.194:51521: BlockingRecv error: recv got EOF from 127.16.90.194:51521 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:40:20.036429 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 25514
I20260926 14:40:20.041502 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 25665
I20260926 14:40:20.046001 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 25799
I20260926 14:40:20.050041 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 25444
2026-09-26T14:40:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17816 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T14:40:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:40:20Z Disabled control of system clock
I20260926 14:40:20.078048 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:36613
--webserver_interface=127.16.90.254
--webserver_port=0
--builtin_ntp_servers=127.16.90.212:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:36613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 14:40:20.188062 25997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:20.188329 25997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:20.188361 25997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:20.192389 25997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:40:20.192442 25997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:20.192456 25997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:20.192472 25997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:40:20.192487 25997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:40:20.197150 25997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:36613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:36613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:20.198185 25997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:20.199546 25997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:40:20.208166 25997 server_base.cc:1034] running on GCE node
W20260926 14:40:20.208200 26003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:20.208200 26002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:20.208227 26005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:20.209046 25997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:20.210117 25997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:20.211359 25997 hybrid_clock.cc:648] HybridClock initialized: now 1790433620211344 us; error 31 us; skew 500 ppm
I20260926 14:40:20.213397 25997 webserver.cc:460] Webserver started at http://127.16.90.254:44099/ using document root <none> and password file <none>
I20260926 14:40:20.213963 25997 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:20.214025 25997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:20.214218 25997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:20.216265 25997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "bd07a390442e4d158fe414beaa80a42f"
format_stamp: "Formatted at 2026-09-26 14:40:20 on dist-test-slave-x4sn"
server_key: "06e38e842adf8bc2251853ff6ccfa9d1"
server_key_iv: "5528d6dbf97c44fe99dfb5972060e36a"
server_key_version: "encryptionkey@0"
I20260926 14:40:20.216778 25997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "bd07a390442e4d158fe414beaa80a42f"
format_stamp: "Formatted at 2026-09-26 14:40:20 on dist-test-slave-x4sn"
server_key: "06e38e842adf8bc2251853ff6ccfa9d1"
server_key_iv: "5528d6dbf97c44fe99dfb5972060e36a"
server_key_version: "encryptionkey@0"
I20260926 14:40:20.220245 25997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:40:20.222502 26011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:20.223824 25997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:40:20.223948 25997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "bd07a390442e4d158fe414beaa80a42f"
format_stamp: "Formatted at 2026-09-26 14:40:20 on dist-test-slave-x4sn"
server_key: "06e38e842adf8bc2251853ff6ccfa9d1"
server_key_iv: "5528d6dbf97c44fe99dfb5972060e36a"
server_key_version: "encryptionkey@0"
I20260926 14:40:20.224076 25997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:20.240706 25997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:20.241370 25997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:20.241540 25997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:20.247841 25997 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:36613
I20260926 14:40:20.247854 26063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:36613 every 8 connection(s)
I20260926 14:40:20.248729 25997 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:40:20.251812 26064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:20.253240 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 25997
I20260926 14:40:20.253348 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:40:20.253626 16747 external_mini_cluster.cc:1428] Setting key 2cc9a4ae00f5a1e80f3279d546e583fb
I20260926 14:40:20.258371 26064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f: Bootstrap starting.
I20260926 14:40:20.260896 26064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:20.261679 26064 log.cc:824] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:40:20.263836 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:20.254977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.16.90.254:36613)
0926 14:40:20.255621 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:20.255642 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:20.255735 (+ 93us) server_negotiation.cc:407] Connection header received
0926 14:40:20.256494 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:20.256518 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:20.257019 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:20.257382 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:20.257818 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.259210 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.260146 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.260817 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.260983 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:20.261888 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:20.261894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:20.261909 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:40:20.262336 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:20.262391 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:40:20.262414 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:20.262512 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:20.262665 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 14:40:20.262926 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":100,"threads_started":1}
I20260926 14:40:20.263919 26064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f: No bootstrap required, opened a new log
I20260926 14:40:20.267612 26064 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd07a390442e4d158fe414beaa80a42f" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36613 } }
I20260926 14:40:20.267876 26064 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:20.267922 26064 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd07a390442e4d158fe414beaa80a42f, State: Initialized, Role: FOLLOWER
I20260926 14:40:20.268298 26064 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd07a390442e4d158fe414beaa80a42f" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36613 } }
I20260926 14:40:20.268407 26064 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:40:20.268460 26064 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:40:20.268551 26064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:20.269447 26064 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd07a390442e4d158fe414beaa80a42f" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36613 } }
I20260926 14:40:20.269763 26064 leader_election.cc:302] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd07a390442e4d158fe414beaa80a42f; no voters:
I20260926 14:40:20.270049 26064 leader_election.cc:288] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:40:20.270191 26069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:40:20.270386 26069 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [term 1 LEADER]: Becoming Leader. State: Replica: bd07a390442e4d158fe414beaa80a42f, State: Running, Role: LEADER
I20260926 14:40:20.270720 26069 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd07a390442e4d158fe414beaa80a42f" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36613 } }
I20260926 14:40:20.271174 26064 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:40:20.272258 26071 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd07a390442e4d158fe414beaa80a42f. Latest consensus state: current_term: 1 leader_uuid: "bd07a390442e4d158fe414beaa80a42f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd07a390442e4d158fe414beaa80a42f" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36613 } } }
I20260926 14:40:20.272270 26070 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd07a390442e4d158fe414beaa80a42f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd07a390442e4d158fe414beaa80a42f" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36613 } } }
I20260926 14:40:20.272444 26071 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [sys.catalog]: This master's current role is: LEADER
I20260926 14:40:20.272481 26070 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f [sys.catalog]: This master's current role is: LEADER
I20260926 14:40:20.272833 26078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:40:20.275578 26078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:40:20.281185 26078 catalog_manager.cc:1347] Generated new cluster ID: 3d857c34d5414a5a9a46abcf1a3db297
I20260926 14:40:20.281261 26078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:40:20.292923 26078 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:40:20.293851 26078 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:40:20.300990 26078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd07a390442e4d158fe414beaa80a42f: Generated new TSK 0
I20260926 14:40:20.301746 26078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:40:20.305550 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:36613
--builtin_ntp_servers=127.16.90.212:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 14:40:20.412732 26088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:20.412966 26088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:20.413000 26088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:20.416914 26088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:20.416960 26088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:20.417016 26088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:40:20.421559 26088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:20.422531 26088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:20.423867 26088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:20.432973 26094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:20.433001 26096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:20.433008 26093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:20.433212 26088 server_base.cc:1034] running on GCE node
I20260926 14:40:20.433596 26088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:20.434183 26088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:20.435328 26088 hybrid_clock.cc:648] HybridClock initialized: now 1790433620435309 us; error 41 us; skew 500 ppm
I20260926 14:40:20.437026 26088 webserver.cc:460] Webserver started at http://127.16.90.193:46387/ using document root <none> and password file <none>
I20260926 14:40:20.437569 26088 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:20.437633 26088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:20.437809 26088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:20.439832 26088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "3a4f8ae8a77749e296d3e4475e1b562e"
format_stamp: "Formatted at 2026-09-26 14:40:20 on dist-test-slave-x4sn"
server_key: "f8708fb9f270d1bc686e8d94e143c4c8"
server_key_iv: "744e5f5875f5b2c9e62dcfe22c5b8a86"
server_key_version: "encryptionkey@0"
I20260926 14:40:20.440338 26088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "3a4f8ae8a77749e296d3e4475e1b562e"
format_stamp: "Formatted at 2026-09-26 14:40:20 on dist-test-slave-x4sn"
server_key: "f8708fb9f270d1bc686e8d94e143c4c8"
server_key_iv: "744e5f5875f5b2c9e62dcfe22c5b8a86"
server_key_version: "encryptionkey@0"
I20260926 14:40:20.443740 26088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:40:20.445998 26102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:20.447075 26088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:40:20.447186 26088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "3a4f8ae8a77749e296d3e4475e1b562e"
format_stamp: "Formatted at 2026-09-26 14:40:20 on dist-test-slave-x4sn"
server_key: "f8708fb9f270d1bc686e8d94e143c4c8"
server_key_iv: "744e5f5875f5b2c9e62dcfe22c5b8a86"
server_key_version: "encryptionkey@0"
I20260926 14:40:20.447274 26088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:20.462119 26088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:20.462783 26088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:20.462949 26088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:20.463503 26088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:20.464430 26088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:20.464476 26088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:20.464527 26088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:20.464555 26088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:20.473114 26088 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:45353
I20260926 14:40:20.473150 26216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:45353 every 8 connection(s)
I20260926 14:40:20.473996 26088 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:40:20.481544 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 26088
I20260926 14:40:20.481639 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:40:20.481925 16747 external_mini_cluster.cc:1428] Setting key d25aa593d85afb964244a7becb69eee2
I20260926 14:40:20.482991 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:20.475680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:36773 (local address 127.16.90.254:36613)
0926 14:40:20.475827 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:20.475830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:20.476860 (+ 1030us) server_negotiation.cc:407] Connection header received
0926 14:40:20.478258 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:20.478261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:20.478324 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:20.478407 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:20.479309 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.479922 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.481074 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.481302 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.481391 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:20.482523 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:20.482526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:20.482529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:40:20.482541 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:20.482565 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:40:20.482575 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:20.482655 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:20.482825 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 14:40:20.482884 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 14:40:20.483949 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:36613
--builtin_ntp_servers=127.16.90.212:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 14:40:20.483983 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:20.476067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:36613 (local address 127.16.90.193:36773)
0926 14:40:20.476705 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:20.476746 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:20.478032 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:20.478586 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:20.478593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:20.478682 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:20.479166 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:20.479179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.480047 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.480051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:20.480951 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:20.480960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.481440 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.481445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:20.481603 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:20.482237 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:40:20.482239 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:20.482281 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:40:20.482360 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:20.482671 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:20.482681 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:20.482687 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:40:20.482817 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 14:40:20.483089 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":111,"threads_started":1}
I20260926 14:40:20.485255 26217 heartbeater.cc:347] Connected to a master server at 127.16.90.254:36613
I20260926 14:40:20.485474 26217 heartbeater.cc:464] Registering TS with master...
I20260926 14:40:20.485950 26217 heartbeater.cc:511] Master 127.16.90.254:36613 requested a full tablet report, sending...
I20260926 14:40:20.487334 26028 ts_manager.cc:194] Registered new tserver with Master: 3a4f8ae8a77749e296d3e4475e1b562e (127.16.90.193:45353)
I20260926 14:40:20.488497 26028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.90.193:36773
W20260926 14:40:20.589149 26220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:20.589380 26220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:20.589412 26220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:20.593294 26220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:20.593340 26220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:20.593389 26220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:40:20.597914 26220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:20.598871 26220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:20.600207 26220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:20.609192 26228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:20.609196 26225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:20.609221 26226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:20.609476 26220 server_base.cc:1034] running on GCE node
I20260926 14:40:20.609910 26220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:20.610483 26220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:20.611706 26220 hybrid_clock.cc:648] HybridClock initialized: now 1790433620611642 us; error 47 us; skew 500 ppm
I20260926 14:40:20.613410 26220 webserver.cc:460] Webserver started at http://127.16.90.194:39425/ using document root <none> and password file <none>
I20260926 14:40:20.613941 26220 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:20.614007 26220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:20.614187 26220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:20.616165 26220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "9ef874a90aa243cfafe51337bd37f669"
format_stamp: "Formatted at 2026-09-26 14:40:20 on dist-test-slave-x4sn"
server_key: "ec18391503b9182f4f394d675a262b68"
server_key_iv: "9d3501291b7d2b46ec3d867981dc925b"
server_key_version: "encryptionkey@0"
I20260926 14:40:20.616681 26220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "9ef874a90aa243cfafe51337bd37f669"
format_stamp: "Formatted at 2026-09-26 14:40:20 on dist-test-slave-x4sn"
server_key: "ec18391503b9182f4f394d675a262b68"
server_key_iv: "9d3501291b7d2b46ec3d867981dc925b"
server_key_version: "encryptionkey@0"
I20260926 14:40:20.620127 26220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:40:20.622567 26234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:20.623718 26220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:40:20.623828 26220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "9ef874a90aa243cfafe51337bd37f669"
format_stamp: "Formatted at 2026-09-26 14:40:20 on dist-test-slave-x4sn"
server_key: "ec18391503b9182f4f394d675a262b68"
server_key_iv: "9d3501291b7d2b46ec3d867981dc925b"
server_key_version: "encryptionkey@0"
I20260926 14:40:20.623917 26220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:20.643422 26220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:20.644137 26220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:20.644307 26220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:20.644871 26220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:20.645776 26220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:20.645823 26220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:20.645874 26220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:20.645900 26220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:20.654153 26220 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:33193
I20260926 14:40:20.654186 26348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:33193 every 8 connection(s)
I20260926 14:40:20.655017 26220 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:40:20.660004 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 26220
I20260926 14:40:20.660112 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:40:20.660388 16747 external_mini_cluster.cc:1428] Setting key c632133f299332056513674d700c0142
I20260926 14:40:20.662483 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:36613
--builtin_ntp_servers=127.16.90.212:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 14:40:20.664618 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:20.656786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:60881 (local address 127.16.90.254:36613)
0926 14:40:20.656973 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:20.656977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:20.657979 (+ 1002us) server_negotiation.cc:407] Connection header received
0926 14:40:20.659362 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:20.659365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:20.659410 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:20.659484 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:20.660470 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.661147 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.662558 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.662853 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.662963 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:20.664204 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:20.664207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:20.664210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:40:20.664221 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:20.664241 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:40:20.664248 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:20.664300 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:20.664452 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 14:40:20.664505 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 14:40:20.665632 26351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:20.657179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:36613 (local address 127.16.90.194:60881)
0926 14:40:20.657832 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:20.657875 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:20.659156 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:20.659640 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:20.659646 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:20.659747 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:20.660317 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:20.660332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.661280 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.661284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:20.662410 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:20.662421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.663040 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.663044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:20.663191 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:20.663837 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:40:20.663839 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:20.663908 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:40:20.664031 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:20.664319 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:20.664326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:20.664332 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:40:20.664461 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 14:40:20.664775 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
I20260926 14:40:20.666787 26349 heartbeater.cc:347] Connected to a master server at 127.16.90.254:36613
I20260926 14:40:20.667040 26349 heartbeater.cc:464] Registering TS with master...
I20260926 14:40:20.667659 26349 heartbeater.cc:511] Master 127.16.90.254:36613 requested a full tablet report, sending...
I20260926 14:40:20.668836 26028 ts_manager.cc:194] Registered new tserver with Master: 9ef874a90aa243cfafe51337bd37f669 (127.16.90.194:33193)
I20260926 14:40:20.669406 26028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.90.194:60881
W20260926 14:40:20.769690 26352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:20.769930 26352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:20.769966 26352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:20.774035 26352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:20.774081 26352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:20.774140 26352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:40:20.778714 26352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:20.779745 26352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:20.781067 26352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:40:20.790046 26352 server_base.cc:1034] running on GCE node
W20260926 14:40:20.789992 26357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:20.790000 26358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:20.790000 26360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:20.790634 26352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:20.791208 26352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:20.792352 26352 hybrid_clock.cc:648] HybridClock initialized: now 1790433620792339 us; error 26 us; skew 500 ppm
I20260926 14:40:20.794049 26352 webserver.cc:460] Webserver started at http://127.16.90.195:34927/ using document root <none> and password file <none>
I20260926 14:40:20.794549 26352 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:20.794605 26352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:20.794762 26352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:20.796702 26352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "d4ade96f9706456ca3d8b6ff639647cf"
format_stamp: "Formatted at 2026-09-26 14:40:20 on dist-test-slave-x4sn"
server_key: "1f2bcf6e7f520fd69473fa42454370e2"
server_key_iv: "8755816b7a4cb9bb879cc61c05c30eb2"
server_key_version: "encryptionkey@0"
I20260926 14:40:20.797209 26352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "d4ade96f9706456ca3d8b6ff639647cf"
format_stamp: "Formatted at 2026-09-26 14:40:20 on dist-test-slave-x4sn"
server_key: "1f2bcf6e7f520fd69473fa42454370e2"
server_key_iv: "8755816b7a4cb9bb879cc61c05c30eb2"
server_key_version: "encryptionkey@0"
I20260926 14:40:20.800621 26352 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:40:20.803179 26366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:20.804454 26352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:40:20.804564 26352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "d4ade96f9706456ca3d8b6ff639647cf"
format_stamp: "Formatted at 2026-09-26 14:40:20 on dist-test-slave-x4sn"
server_key: "1f2bcf6e7f520fd69473fa42454370e2"
server_key_iv: "8755816b7a4cb9bb879cc61c05c30eb2"
server_key_version: "encryptionkey@0"
I20260926 14:40:20.804669 26352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:20.834856 26352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:20.835596 26352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:20.835795 26352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:20.836386 26352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:20.837327 26352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:20.837373 26352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:20.837426 26352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:20.837455 26352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:20.846012 26352 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:42981
I20260926 14:40:20.846047 26480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:42981 every 8 connection(s)
I20260926 14:40:20.846907 26352 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:40:20.849264 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 26352
I20260926 14:40:20.849377 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:40:20.849701 16747 external_mini_cluster.cc:1428] Setting key 3501e544557825fcbe59d0686f695ac8
I20260926 14:40:20.855962 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:20.848769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:45461 (local address 127.16.90.254:36613)
0926 14:40:20.848909 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:20.848913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:20.850089 (+ 1176us) server_negotiation.cc:407] Connection header received
0926 14:40:20.851713 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:20.851716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:20.851767 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:20.851841 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:20.852657 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.853176 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.854227 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.854450 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.854547 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:20.855562 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:20.855565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:20.855568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:40:20.855580 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:20.855600 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:40:20.855609 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:20.855686 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:20.855797 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 14:40:20.855849 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:40:20.856827 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:20.849175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:36613 (local address 127.16.90.195:45461)
0926 14:40:20.849922 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:20.849964 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:20.851497 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:20.851978 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:20.851985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:20.852069 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:20.852550 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:20.852562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.853261 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.853263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:20.854118 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:20.854125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:20.854562 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:20.854565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:20.854710 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:20.855304 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:40:20.855306 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:20.855346 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:40:20.855409 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:20.855696 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:20.855700 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:20.855704 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:40:20.855789 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:40:20.856008 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":141,"threads_started":1}
I20260926 14:40:20.857877 26481 heartbeater.cc:347] Connected to a master server at 127.16.90.254:36613
I20260926 14:40:20.858080 26481 heartbeater.cc:464] Registering TS with master...
I20260926 14:40:20.858568 26481 heartbeater.cc:511] Master 127.16.90.254:36613 requested a full tablet report, sending...
I20260926 14:40:20.859513 26028 ts_manager.cc:194] Registered new tserver with Master: d4ade96f9706456ca3d8b6ff639647cf (127.16.90.195:42981)
I20260926 14:40:20.860214 26028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.90.195:45461
I20260926 14:40:20.863550 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 14:40:20.866230 26067 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:20.865425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.16.90.254:36613)
0926 14:40:20.865555 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:20.865559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:20.865678 (+ 119us) server_negotiation.cc:407] Connection header received
0926 14:40:20.865771 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:20.865773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:20.865809 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:20.865870 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:20.866134 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37780: BlockingRecv error: recv got EOF from 127.0.0.1:37780 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:40:20.866448 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 26088
I20260926 14:40:20.871977 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 26220
I20260926 14:40:20.876875 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 26352
I20260926 14:40:20.881075 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 25997
2026-09-26T14:40:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (829 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T14:40:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:40:20Z Disabled control of system clock
I20260926 14:40:20.907338 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:41855
--webserver_interface=127.16.90.254
--webserver_port=0
--builtin_ntp_servers=127.16.90.212:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:41855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 14:40:21.011926 26492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:21.012188 26492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:21.012225 26492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:21.016098 26492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:40:21.016144 26492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:21.016158 26492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:21.016178 26492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:40:21.016192 26492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:40:21.020781 26492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:41855
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:41855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:21.021814 26492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:21.023099 26492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:21.031085 26497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:21.031175 26492 server_base.cc:1034] running on GCE node
W20260926 14:40:21.031191 26500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:21.031056 26498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:21.031922 26492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:21.032943 26492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:21.034085 26492 hybrid_clock.cc:648] HybridClock initialized: now 1790433621034062 us; error 38 us; skew 500 ppm
I20260926 14:40:21.035830 26492 webserver.cc:460] Webserver started at http://127.16.90.254:44671/ using document root <none> and password file <none>
I20260926 14:40:21.036355 26492 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:21.036413 26492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:21.036584 26492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:21.038497 26492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "cd4daef2e97f40fd9c287a94de71694c"
format_stamp: "Formatted at 2026-09-26 14:40:21 on dist-test-slave-x4sn"
server_key: "884fdb8d29c56dae84442b1fd7856a33"
server_key_iv: "0c3efebc8cef7d5972755546e53d6005"
server_key_version: "encryptionkey@0"
I20260926 14:40:21.039047 26492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "cd4daef2e97f40fd9c287a94de71694c"
format_stamp: "Formatted at 2026-09-26 14:40:21 on dist-test-slave-x4sn"
server_key: "884fdb8d29c56dae84442b1fd7856a33"
server_key_iv: "0c3efebc8cef7d5972755546e53d6005"
server_key_version: "encryptionkey@0"
I20260926 14:40:21.042527 26492 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 14:40:21.044847 26506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:21.046137 26492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:40:21.046247 26492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "cd4daef2e97f40fd9c287a94de71694c"
format_stamp: "Formatted at 2026-09-26 14:40:21 on dist-test-slave-x4sn"
server_key: "884fdb8d29c56dae84442b1fd7856a33"
server_key_iv: "0c3efebc8cef7d5972755546e53d6005"
server_key_version: "encryptionkey@0"
I20260926 14:40:21.046341 26492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:21.050665 26492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:21.051227 26492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:21.051378 26492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:21.057926 26492 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:41855
I20260926 14:40:21.057945 26558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:41855 every 8 connection(s)
I20260926 14:40:21.058794 26492 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:40:21.062003 26559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:21.062288 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 26492
I20260926 14:40:21.062378 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:40:21.062629 16747 external_mini_cluster.cc:1428] Setting key a265f1a703ef4784ae6e0135fdaf4019
I20260926 14:40:21.068521 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:21.064020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.16.90.254:41855)
0926 14:40:21.064721 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:21.064744 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:21.064787 (+ 43us) server_negotiation.cc:407] Connection header received
0926 14:40:21.065503 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:21.065535 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:21.065985 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:21.066347 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:21.067048 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:21.067053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:21.067068 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:40:21.067078 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:21.067144 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:40:21.067165 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:21.067303 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:21.067373 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 14:40:21.067628 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":124,"threads_started":1}
I20260926 14:40:21.069239 26559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c: Bootstrap starting.
I20260926 14:40:21.071575 26559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:21.072319 26559 log.cc:824] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:40:21.074265 26559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c: No bootstrap required, opened a new log
I20260926 14:40:21.077617 26559 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4daef2e97f40fd9c287a94de71694c" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41855 } }
I20260926 14:40:21.077847 26559 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:21.077905 26559 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd4daef2e97f40fd9c287a94de71694c, State: Initialized, Role: FOLLOWER
I20260926 14:40:21.078325 26559 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4daef2e97f40fd9c287a94de71694c" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41855 } }
I20260926 14:40:21.078473 26559 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:40:21.078543 26559 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:40:21.078620 26559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:21.079420 26559 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4daef2e97f40fd9c287a94de71694c" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41855 } }
I20260926 14:40:21.079735 26559 leader_election.cc:302] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd4daef2e97f40fd9c287a94de71694c; no voters:
I20260926 14:40:21.079991 26559 leader_election.cc:288] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:40:21.080070 26564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:40:21.080294 26564 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [term 1 LEADER]: Becoming Leader. State: Replica: cd4daef2e97f40fd9c287a94de71694c, State: Running, Role: LEADER
I20260926 14:40:21.080577 26564 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4daef2e97f40fd9c287a94de71694c" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41855 } }
I20260926 14:40:21.081075 26559 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:40:21.082021 26566 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd4daef2e97f40fd9c287a94de71694c. Latest consensus state: current_term: 1 leader_uuid: "cd4daef2e97f40fd9c287a94de71694c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4daef2e97f40fd9c287a94de71694c" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41855 } } }
I20260926 14:40:21.082155 26566 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [sys.catalog]: This master's current role is: LEADER
I20260926 14:40:21.082433 26565 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd4daef2e97f40fd9c287a94de71694c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4daef2e97f40fd9c287a94de71694c" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 41855 } } }
I20260926 14:40:21.082527 26573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:40:21.082547 26565 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c [sys.catalog]: This master's current role is: LEADER
I20260926 14:40:21.085644 26573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:40:21.091080 26573 catalog_manager.cc:1347] Generated new cluster ID: 67a23e76ccec4a80803306c8c2a118ff
I20260926 14:40:21.091135 26573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:40:21.103657 26573 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:40:21.103974 26573 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:40:21.109212 26573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd4daef2e97f40fd9c287a94de71694c: Generated new TSK 0
I20260926 14:40:21.109872 26573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:40:21.120599 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:41855
--builtin_ntp_servers=127.16.90.212:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 14:40:21.228658 26583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:21.228899 26583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:21.228941 26583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:21.232914 26583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:21.232970 26583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:21.233044 26583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:40:21.237717 26583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:41855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:21.238744 26583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:21.240200 26583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:21.249243 26591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:21.249254 26588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:21.249362 26583 server_base.cc:1034] running on GCE node
W20260926 14:40:21.249485 26589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:21.249835 26583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:21.250460 26583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:21.251643 26583 hybrid_clock.cc:648] HybridClock initialized: now 1790433621251609 us; error 29 us; skew 500 ppm
I20260926 14:40:21.253394 26583 webserver.cc:460] Webserver started at http://127.16.90.193:46349/ using document root <none> and password file <none>
I20260926 14:40:21.253947 26583 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:21.254012 26583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:21.254227 26583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:21.256307 26583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "64ea3d310d1c43bdaac6b491578cb787"
format_stamp: "Formatted at 2026-09-26 14:40:21 on dist-test-slave-x4sn"
server_key: "8f5c0a73b5ff2af7aa20722a430ad453"
server_key_iv: "e2f717ea2de2960bcb2d6496985c3b83"
server_key_version: "encryptionkey@0"
I20260926 14:40:21.256840 26583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "64ea3d310d1c43bdaac6b491578cb787"
format_stamp: "Formatted at 2026-09-26 14:40:21 on dist-test-slave-x4sn"
server_key: "8f5c0a73b5ff2af7aa20722a430ad453"
server_key_iv: "e2f717ea2de2960bcb2d6496985c3b83"
server_key_version: "encryptionkey@0"
I20260926 14:40:21.260443 26583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:40:21.262879 26597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:21.264117 26583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 14:40:21.264228 26583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "64ea3d310d1c43bdaac6b491578cb787"
format_stamp: "Formatted at 2026-09-26 14:40:21 on dist-test-slave-x4sn"
server_key: "8f5c0a73b5ff2af7aa20722a430ad453"
server_key_iv: "e2f717ea2de2960bcb2d6496985c3b83"
server_key_version: "encryptionkey@0"
I20260926 14:40:21.264324 26583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:21.268700 26583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:21.269268 26583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:21.269419 26583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:21.269923 26583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:21.270795 26583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:21.270843 26583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:21.270891 26583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:21.270918 26583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:21.279405 26583 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:35323
I20260926 14:40:21.279424 26710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:35323 every 8 connection(s)
I20260926 14:40:21.280328 26583 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:40:21.286325 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:21.282203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:35157 (local address 127.16.90.254:41855)
0926 14:40:21.282367 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:21.282373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:21.283454 (+ 1081us) server_negotiation.cc:407] Connection header received
0926 14:40:21.284787 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:21.284789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:21.284826 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:21.284888 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:21.285947 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:21.285949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:21.285951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:40:21.285962 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:21.285984 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:40:21.285992 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:21.286064 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:21.286157 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 14:40:21.286194 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:40:21.286499 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 26583
I20260926 14:40:21.286607 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:40:21.286877 16747 external_mini_cluster.cc:1428] Setting key a57620599fd500dd800a58006920fe79
I20260926 14:40:21.287215 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:21.282608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:41855 (local address 127.16.90.193:35157)
0926 14:40:21.283308 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:21.283361 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:21.284580 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:21.285088 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:21.285095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:21.285202 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:21.285696 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:40:21.285698 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:21.285737 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:40:21.285804 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:21.286054 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:21.286059 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:21.286063 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:40:21.286185 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 14:40:21.286373 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260926 14:40:21.288378 26711 heartbeater.cc:347] Connected to a master server at 127.16.90.254:41855
I20260926 14:40:21.288584 26711 heartbeater.cc:464] Registering TS with master...
I20260926 14:40:21.288719 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:41855
--builtin_ntp_servers=127.16.90.212:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 14:40:21.288986 26711 heartbeater.cc:511] Master 127.16.90.254:41855 requested a full tablet report, sending...
I20260926 14:40:21.290349 26522 ts_manager.cc:194] Registered new tserver with Master: 64ea3d310d1c43bdaac6b491578cb787 (127.16.90.193:35323)
W20260926 14:40:21.394068 26714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:21.394315 26714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:21.394367 26714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:21.398279 26714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:21.398336 26714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:21.398411 26714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:40:21.403020 26714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:41855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:21.404086 26714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:21.405421 26714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:21.414731 26722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:21.414772 26720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:21.414870 26714 server_base.cc:1034] running on GCE node
W20260926 14:40:21.414772 26719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:21.415397 26714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:21.416074 26714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:21.417227 26714 hybrid_clock.cc:648] HybridClock initialized: now 1790433621417210 us; error 30 us; skew 500 ppm
I20260926 14:40:21.418885 26714 webserver.cc:460] Webserver started at http://127.16.90.194:43661/ using document root <none> and password file <none>
I20260926 14:40:21.419428 26714 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:21.419493 26714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:21.419713 26714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:21.421672 26714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "112f73d6fb0d44ff8ce0d642fafc334d"
format_stamp: "Formatted at 2026-09-26 14:40:21 on dist-test-slave-x4sn"
server_key: "a22f29d3194f2e23d45ec8b043007eef"
server_key_iv: "f7628849fe5c0749a82ef5e8ab8194db"
server_key_version: "encryptionkey@0"
I20260926 14:40:21.422204 26714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "112f73d6fb0d44ff8ce0d642fafc334d"
format_stamp: "Formatted at 2026-09-26 14:40:21 on dist-test-slave-x4sn"
server_key: "a22f29d3194f2e23d45ec8b043007eef"
server_key_iv: "f7628849fe5c0749a82ef5e8ab8194db"
server_key_version: "encryptionkey@0"
I20260926 14:40:21.425743 26714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:40:21.428190 26728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:21.429455 26714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:40:21.429595 26714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "112f73d6fb0d44ff8ce0d642fafc334d"
format_stamp: "Formatted at 2026-09-26 14:40:21 on dist-test-slave-x4sn"
server_key: "a22f29d3194f2e23d45ec8b043007eef"
server_key_iv: "f7628849fe5c0749a82ef5e8ab8194db"
server_key_version: "encryptionkey@0"
I20260926 14:40:21.429703 26714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:21.434408 26714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:21.434994 26714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:21.435166 26714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:21.435737 26714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:21.436625 26714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:21.436684 26714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:21.436739 26714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:21.436769 26714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:21.445480 26714 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:43655
I20260926 14:40:21.445533 26842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:43655 every 8 connection(s)
I20260926 14:40:21.446372 26714 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:40:21.452054 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:21.448107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:57005 (local address 127.16.90.254:41855)
0926 14:40:21.448259 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:21.448262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:21.449239 (+ 977us) server_negotiation.cc:407] Connection header received
0926 14:40:21.450511 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:21.450515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:21.450559 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:21.450630 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:21.451584 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:21.451587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:21.451590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:40:21.451602 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:21.451623 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:40:21.451631 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:21.451734 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:21.451873 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 14:40:21.451916 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 14:40:21.452848 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:21.448465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:41855 (local address 127.16.90.194:57005)
0926 14:40:21.449080 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:21.449132 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:21.450318 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:21.450786 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:21.450792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:21.450879 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:21.451337 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:40:21.451339 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:21.451378 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:40:21.451449 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:21.451736 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:21.451744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:21.451750 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:40:21.451904 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 14:40:21.452136 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260926 14:40:21.453855 26843 heartbeater.cc:347] Connected to a master server at 127.16.90.254:41855
I20260926 14:40:21.454051 26843 heartbeater.cc:464] Registering TS with master...
I20260926 14:40:21.454332 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 26714
I20260926 14:40:21.454439 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:40:21.454458 26843 heartbeater.cc:511] Master 127.16.90.254:41855 requested a full tablet report, sending...
I20260926 14:40:21.454710 16747 external_mini_cluster.cc:1428] Setting key 880503f933650409fe74e29a692a54c5
I20260926 14:40:21.455410 26522 ts_manager.cc:194] Registered new tserver with Master: 112f73d6fb0d44ff8ce0d642fafc334d (127.16.90.194:43655)
I20260926 14:40:21.456439 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:41855
--builtin_ntp_servers=127.16.90.212:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 14:40:21.561260 26846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:21.561503 26846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:21.561556 26846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:21.565800 26846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:21.565850 26846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:21.565927 26846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:40:21.570658 26846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:41855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:21.571653 26846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:21.573215 26846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:21.582249 26852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:21.582249 26855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:21.582253 26853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:21.582548 26846 server_base.cc:1034] running on GCE node
I20260926 14:40:21.582903 26846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:21.583514 26846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:21.584671 26846 hybrid_clock.cc:648] HybridClock initialized: now 1790433621584653 us; error 33 us; skew 500 ppm
I20260926 14:40:21.586411 26846 webserver.cc:460] Webserver started at http://127.16.90.195:42899/ using document root <none> and password file <none>
I20260926 14:40:21.586963 26846 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:21.587023 26846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:21.587194 26846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:21.589208 26846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "81b83b5a1a2d4357bdbae2ec5c6cf225"
format_stamp: "Formatted at 2026-09-26 14:40:21 on dist-test-slave-x4sn"
server_key: "f3ce7d3a1404b162128c2eef3fac77cb"
server_key_iv: "d4d02b18c6a12ddc60572992051c8d05"
server_key_version: "encryptionkey@0"
I20260926 14:40:21.589716 26846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "81b83b5a1a2d4357bdbae2ec5c6cf225"
format_stamp: "Formatted at 2026-09-26 14:40:21 on dist-test-slave-x4sn"
server_key: "f3ce7d3a1404b162128c2eef3fac77cb"
server_key_iv: "d4d02b18c6a12ddc60572992051c8d05"
server_key_version: "encryptionkey@0"
I20260926 14:40:21.593245 26846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 14:40:21.595568 26861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:21.596728 26846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:40:21.596828 26846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "81b83b5a1a2d4357bdbae2ec5c6cf225"
format_stamp: "Formatted at 2026-09-26 14:40:21 on dist-test-slave-x4sn"
server_key: "f3ce7d3a1404b162128c2eef3fac77cb"
server_key_iv: "d4d02b18c6a12ddc60572992051c8d05"
server_key_version: "encryptionkey@0"
I20260926 14:40:21.596910 26846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:21.601178 26846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:21.601742 26846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:21.601871 26846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:21.602339 26846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:21.603134 26846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:21.603188 26846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:21.603223 26846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:21.603242 26846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:21.611279 26846 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:38639
I20260926 14:40:21.611320 26974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:38639 every 8 connection(s)
I20260926 14:40:21.612192 26846 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:40:21.617699 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:21.613844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:42139 (local address 127.16.90.254:41855)
0926 14:40:21.613975 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:21.613979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:21.614999 (+ 1020us) server_negotiation.cc:407] Connection header received
0926 14:40:21.616214 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:21.616217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:21.616254 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:21.616327 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:21.617276 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:21.617278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:21.617280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:40:21.617287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:21.617302 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:40:21.617308 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:21.617370 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:21.617551 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 14:40:21.617586 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:40:21.618508 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:21.614241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:41855 (local address 127.16.90.195:42139)
0926 14:40:21.614857 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:21.614895 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:21.616042 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:21.616463 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:21.616469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:21.616552 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:21.617062 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:40:21.617063 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:21.617102 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:40:21.617166 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:21.617426 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:21.617434 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:21.617440 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:40:21.617592 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 14:40:21.617797 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
I20260926 14:40:21.619632 26975 heartbeater.cc:347] Connected to a master server at 127.16.90.254:41855
I20260926 14:40:21.619855 26975 heartbeater.cc:464] Registering TS with master...
I20260926 14:40:21.620227 26975 heartbeater.cc:511] Master 127.16.90.254:41855 requested a full tablet report, sending...
I20260926 14:40:21.621168 26522 ts_manager.cc:194] Registered new tserver with Master: 81b83b5a1a2d4357bdbae2ec5c6cf225 (127.16.90.195:38639)
I20260926 14:40:21.621694 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 26846
I20260926 14:40:21.621810 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:40:21.622105 16747 external_mini_cluster.cc:1428] Setting key d9e457103e2e9b4838a604c515865de1
I20260926 14:40:21.623899 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 14:40:21.626245 26562 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:40:21.625544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.16.90.254:41855)
0926 14:40:21.625676 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:21.625680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:21.625795 (+ 115us) server_negotiation.cc:407] Connection header received
0926 14:40:21.625883 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:21.625885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:21.625915 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:21.625975 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:21.626126 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35966: BlockingRecv error: recv got EOF from 127.0.0.1:35966 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:40:21.626394 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 26583
I20260926 14:40:21.631660 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 26714
I20260926 14:40:21.635726 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 26846
I20260926 14:40:21.640075 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 26492
2026-09-26T14:40:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (759 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:40:21 dist-test-slave-x4sn krb5kdc[26985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:40:21 dist-test-slave-x4sn krb5kdc[26985](info): set up 2 sockets
Sep 26 14:40:21 dist-test-slave-x4sn krb5kdc[26985](info): commencing operation
krb5kdc: starting...
W20260926 14:40:22.605933 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:40:22 dist-test-slave-x4sn krb5kdc[26985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:40:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:40:22.720088 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35861
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:35861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:40:22.826843 26999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:22.827090 26999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:22.827129 26999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:22.831022 26999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:40:22.831108 26999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:22.831127 26999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:22.831142 26999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:40:22.831156 26999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:40:22.836576 26999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:35861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:22.837570 26999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:22.838541 26999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:22.846473 27004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:22.846487 27007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:22.846487 27005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:22.846670 26999 server_base.cc:1034] running on GCE node
I20260926 14:40:22.847224 26999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:22.848289 26999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:22.849473 26999 hybrid_clock.cc:648] HybridClock initialized: now 1790433622849457 us; error 26 us; skew 500 ppm
Sep 26 14:40:22 dist-test-slave-x4sn krb5kdc[26985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:40:22.852061 26999 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:40:22.852855 26999 webserver.cc:460] Webserver started at http://127.16.90.254:36207/ using document root <none> and password file <none>
I20260926 14:40:22.853351 26999 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:22.853406 26999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:22.853564 26999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:22.855479 26999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "f23f3bf76727446e8db9e0b30c916847"
format_stamp: "Formatted at 2026-09-26 14:40:22 on dist-test-slave-x4sn"
server_key: "85f9c64ec03ed8efcc5a665f3c73609a"
server_key_iv: "be20d88ad26f59c483d646032447cb5f"
server_key_version: "encryptionkey@0"
I20260926 14:40:22.856029 26999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "f23f3bf76727446e8db9e0b30c916847"
format_stamp: "Formatted at 2026-09-26 14:40:22 on dist-test-slave-x4sn"
server_key: "85f9c64ec03ed8efcc5a665f3c73609a"
server_key_iv: "be20d88ad26f59c483d646032447cb5f"
server_key_version: "encryptionkey@0"
I20260926 14:40:22.859496 26999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:40:22.861790 27014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:22.862943 26999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:40:22.863044 26999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "f23f3bf76727446e8db9e0b30c916847"
format_stamp: "Formatted at 2026-09-26 14:40:22 on dist-test-slave-x4sn"
server_key: "85f9c64ec03ed8efcc5a665f3c73609a"
server_key_iv: "be20d88ad26f59c483d646032447cb5f"
server_key_version: "encryptionkey@0"
I20260926 14:40:22.863138 26999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:22.881794 26999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:22.885059 26999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:22.885250 26999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:22.891839 26999 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:35861
I20260926 14:40:22.892068 27066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:35861 every 8 connection(s)
I20260926 14:40:22.892791 26999 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:40:22.895483 27067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:22.895915 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 26999
I20260926 14:40:22.896029 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:40:22.896292 16747 external_mini_cluster.cc:1428] Setting key afd3ec64ea14f2c5e6704c7516594ab0
I20260926 14:40:22.902413 27067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847: Bootstrap starting.
I20260926 14:40:22.905027 27067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847: Neither blocks nor log segments found. Creating new log.
Sep 26 14:40:22 dist-test-slave-x4sn krb5kdc[26985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:40:22.905880 27067 log.cc:824] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847: Log is configured to *not* fsync() on all Append() calls
I20260926 14:40:22.907958 27067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847: No bootstrap required, opened a new log
I20260926 14:40:22.910080 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:22.897682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.16.90.254:35861)
0926 14:40:22.898088 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:22.898103 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:22.898137 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:40:22.898814 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:22.898831 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:22.899167 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:22.899444 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:22.900343 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:22.901318 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:22.902445 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:22.902956 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:22.903070 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:22.905707 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:22.905713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:22.905725 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:40:22.905752 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:22.907385 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:22.907874 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:22.907878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:22.907883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:22.907989 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:22.908259 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:22.908261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:22.908262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:22.908633 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:22.908835 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:22.909098 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 14:40:22.909373 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260926 14:40:22.911631 27067 raft_consensus.cc:348] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23f3bf76727446e8db9e0b30c916847" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35861 } }
I20260926 14:40:22.911943 27067 raft_consensus.cc:374] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:22.912000 27067 raft_consensus.cc:729] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f3bf76727446e8db9e0b30c916847, State: Initialized, Role: FOLLOWER
I20260926 14:40:22.912452 27067 consensus_queue.cc:260] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23f3bf76727446e8db9e0b30c916847" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35861 } }
I20260926 14:40:22.912599 27067 raft_consensus.cc:388] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:40:22.912649 27067 raft_consensus.cc:482] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:40:22.912720 27067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:22.913774 27067 raft_consensus.cc:504] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23f3bf76727446e8db9e0b30c916847" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35861 } }
I20260926 14:40:22.914069 27067 leader_election.cc:302] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f23f3bf76727446e8db9e0b30c916847; no voters:
I20260926 14:40:22.914321 27067 leader_election.cc:288] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:40:22.914438 27072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:40:22.914616 27072 raft_consensus.cc:686] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [term 1 LEADER]: Becoming Leader. State: Replica: f23f3bf76727446e8db9e0b30c916847, State: Running, Role: LEADER
I20260926 14:40:22.914930 27072 consensus_queue.cc:237] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23f3bf76727446e8db9e0b30c916847" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35861 } }
I20260926 14:40:22.915452 27067 sys_catalog.cc:564] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:40:22.916498 27074 sys_catalog.cc:455] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f23f3bf76727446e8db9e0b30c916847. Latest consensus state: current_term: 1 leader_uuid: "f23f3bf76727446e8db9e0b30c916847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23f3bf76727446e8db9e0b30c916847" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35861 } } }
I20260926 14:40:22.916652 27074 sys_catalog.cc:458] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [sys.catalog]: This master's current role is: LEADER
I20260926 14:40:22.917021 27081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:40:22.917419 27073 sys_catalog.cc:455] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f23f3bf76727446e8db9e0b30c916847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23f3bf76727446e8db9e0b30c916847" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35861 } } }
I20260926 14:40:22.917517 27073 sys_catalog.cc:458] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847 [sys.catalog]: This master's current role is: LEADER
I20260926 14:40:22.920221 27081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:40:22.926047 27081 catalog_manager.cc:1347] Generated new cluster ID: adbb187489484cdf93ca2d78bc3dacac
I20260926 14:40:22.926118 27081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:40:22.936669 27081 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:40:22.937928 27081 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:40:22.948576 27081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847: Generated new TSK 0
I20260926 14:40:22.949450 27081 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:40:22.979784 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35861
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:40:23.089787 27093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:23.090047 27093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:23.090106 27093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:23.094146 27093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:23.094213 27093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:23.094293 27093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:40:23.099466 27093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:23.100535 27093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:23.101544 27093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:23.111008 27101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:23.111016 27099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:23.111198 27093 server_base.cc:1034] running on GCE node
W20260926 14:40:23.111016 27098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:23.111768 27093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:23.112401 27093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:23.113580 27093 hybrid_clock.cc:648] HybridClock initialized: now 1790433623113555 us; error 35 us; skew 500 ppm
Sep 26 14:40:23 dist-test-slave-x4sn krb5kdc[26985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:40:23.116340 27093 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:40:23.117277 27093 webserver.cc:460] Webserver started at http://127.16.90.193:36043/ using document root <none> and password file <none>
I20260926 14:40:23.117861 27093 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:23.117923 27093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:23.118090 27093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:23.120088 27093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40"
format_stamp: "Formatted at 2026-09-26 14:40:23 on dist-test-slave-x4sn"
server_key: "a3c24e0de4bbf64d202d21480fdf35d9"
server_key_iv: "cc597646c504eea32b554b570f09c8dd"
server_key_version: "encryptionkey@0"
I20260926 14:40:23.120576 27093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40"
format_stamp: "Formatted at 2026-09-26 14:40:23 on dist-test-slave-x4sn"
server_key: "a3c24e0de4bbf64d202d21480fdf35d9"
server_key_iv: "cc597646c504eea32b554b570f09c8dd"
server_key_version: "encryptionkey@0"
I20260926 14:40:23.124101 27093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:40:23.126405 27108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:23.127568 27093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:40:23.127714 27093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40"
format_stamp: "Formatted at 2026-09-26 14:40:23 on dist-test-slave-x4sn"
server_key: "a3c24e0de4bbf64d202d21480fdf35d9"
server_key_iv: "cc597646c504eea32b554b570f09c8dd"
server_key_version: "encryptionkey@0"
I20260926 14:40:23.127815 27093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:23.141335 27093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:23.144199 27093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:23.144351 27093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:23.144878 27093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:23.145684 27093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:23.145731 27093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:23.145771 27093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:23.145784 27093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:23.153697 27093 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:46655
I20260926 14:40:23.153723 27222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:46655 every 8 connection(s)
I20260926 14:40:23.154547 27093 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:40:23.155951 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 27093
I20260926 14:40:23.156085 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:40:23.156436 16747 external_mini_cluster.cc:1428] Setting key 89e86427ce91dc670a070b6225f51ff3
Sep 26 14:40:23 dist-test-slave-x4sn krb5kdc[26985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
I20260926 14:40:23.169153 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.156428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:53897 (local address 127.16.90.254:35861)
0926 14:40:23.156567 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:23.156571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:23.157649 (+ 1078us) server_negotiation.cc:407] Connection header received
0926 14:40:23.159066 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:23.159070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:23.159130 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:23.159211 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:23.160582 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.161337 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.162525 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.162903 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.163013 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.165562 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:23.165565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:23.165567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:40:23.165594 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:23.167020 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:23.167556 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:23.167559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:23.167560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:23.167642 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:23.167988 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:23.167992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:23.167993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:23.168202 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:23.168281 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:23.168920 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 14:40:23.169029 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 14:40:23.170321 27225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.156964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35861 (local address 127.16.90.193:53897)
0926 14:40:23.157497 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:23.157549 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:23.158831 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:23.159530 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:23.159537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:23.159942 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:23.160426 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.160440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.161443 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.161446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.162398 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.162408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.163040 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.163042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.163190 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.163823 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:23.163839 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:23.165378 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:23.167133 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:23.167139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:23.167154 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:23.167467 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:23.167751 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:23.167753 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:23.167754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:23.167898 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:23.168333 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:23.168340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:23.168668 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 14:40:23.168875 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 14:40:23.169212 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":145,"threads_started":1}
I20260926 14:40:23.171545 27223 heartbeater.cc:347] Connected to a master server at 127.16.90.254:35861
I20260926 14:40:23.171793 27223 heartbeater.cc:464] Registering TS with master...
I20260926 14:40:23.172263 27223 heartbeater.cc:511] Master 127.16.90.254:35861 requested a full tablet report, sending...
I20260926 14:40:23.173717 27031 ts_manager.cc:194] Registered new tserver with Master: 2b9c144e14bc4aeebe9cf1ba2aec2d40 (127.16.90.193:46655)
I20260926 14:40:23.174798 27031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:53897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:40:23.182482 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35861
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:40:23.288726 27228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:23.288981 27228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:23.289022 27228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:23.292943 27228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:23.293002 27228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:23.293058 27228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:40:23.298143 27228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:23.299126 27228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:23.300150 27228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:23.309087 27234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:23.309118 27236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:23.309137 27233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:23.309239 27228 server_base.cc:1034] running on GCE node
I20260926 14:40:23.309707 27228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:23.310284 27228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:23.311439 27228 hybrid_clock.cc:648] HybridClock initialized: now 1790433623311416 us; error 33 us; skew 500 ppm
Sep 26 14:40:23 dist-test-slave-x4sn krb5kdc[26985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:40:23.314114 27228 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:40:23.314961 27228 webserver.cc:460] Webserver started at http://127.16.90.194:33467/ using document root <none> and password file <none>
I20260926 14:40:23.315475 27228 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:23.315542 27228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:23.315770 27228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:23.317849 27228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "bc7670bc20254a71b6c1ec14e83f58cd"
format_stamp: "Formatted at 2026-09-26 14:40:23 on dist-test-slave-x4sn"
server_key: "3d4230b6ca4c1baf56b597f5e1f3e8f7"
server_key_iv: "04cebf1f80abef648c9232c2a7e612b5"
server_key_version: "encryptionkey@0"
I20260926 14:40:23.318360 27228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "bc7670bc20254a71b6c1ec14e83f58cd"
format_stamp: "Formatted at 2026-09-26 14:40:23 on dist-test-slave-x4sn"
server_key: "3d4230b6ca4c1baf56b597f5e1f3e8f7"
server_key_iv: "04cebf1f80abef648c9232c2a7e612b5"
server_key_version: "encryptionkey@0"
I20260926 14:40:23.322059 27228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 14:40:23.324383 27243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:23.325531 27228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:40:23.325627 27228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "bc7670bc20254a71b6c1ec14e83f58cd"
format_stamp: "Formatted at 2026-09-26 14:40:23 on dist-test-slave-x4sn"
server_key: "3d4230b6ca4c1baf56b597f5e1f3e8f7"
server_key_iv: "04cebf1f80abef648c9232c2a7e612b5"
server_key_version: "encryptionkey@0"
I20260926 14:40:23.325739 27228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:23.341826 27228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:23.344894 27228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:23.345064 27228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:23.345618 27228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:23.346400 27228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:23.346446 27228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:23.346485 27228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:23.346514 27228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:23.354803 27228 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:41927
I20260926 14:40:23.354837 27357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:41927 every 8 connection(s)
I20260926 14:40:23.355656 27228 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:40:23.360513 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 27228
I20260926 14:40:23.360648 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:40:23.360947 16747 external_mini_cluster.cc:1428] Setting key 17681a9ce06631857c9fbddfcbd9c2dd
Sep 26 14:40:23 dist-test-slave-x4sn krb5kdc[26985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:40:23.369566 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.357408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:51879 (local address 127.16.90.254:35861)
0926 14:40:23.357549 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:23.357553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:23.358365 (+ 812us) server_negotiation.cc:407] Connection header received
0926 14:40:23.359722 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:23.359725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:23.359780 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:23.359859 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:23.361287 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.361855 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.363090 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.363399 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.363504 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.365978 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:23.365981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:23.365983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:40:23.366006 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:23.367383 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:23.368073 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:23.368076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:23.368077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:23.368157 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:23.368517 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:23.368518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:23.368520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:23.368697 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:23.368767 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:23.369345 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 14:40:23.369449 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
I20260926 14:40:23.370754 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.357757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35861 (local address 127.16.90.194:51879)
0926 14:40:23.358215 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:23.358257 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:23.359512 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:23.360025 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:23.360032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:23.360455 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:23.361112 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.361132 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.361980 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.361983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.362964 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.362974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.363548 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.363552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.363758 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.364347 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:23.364363 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:23.365828 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:23.367514 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:23.367521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:23.367561 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:23.367963 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:23.368243 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:23.368246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:23.368248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:23.368435 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:23.368788 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:23.368795 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:23.369109 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 14:40:23.369326 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 14:40:23.369677 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260926 14:40:23.371878 27358 heartbeater.cc:347] Connected to a master server at 127.16.90.254:35861
I20260926 14:40:23.372090 27358 heartbeater.cc:464] Registering TS with master...
I20260926 14:40:23.372548 27358 heartbeater.cc:511] Master 127.16.90.254:35861 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
I20260926 14:40:23.373564 27031 ts_manager.cc:194] Registered new tserver with Master: bc7670bc20254a71b6c1ec14e83f58cd (127.16.90.194:41927)
I20260926 14:40:23.374150 27031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:51879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:40:23.389029 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35861
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:40:23.496878 27363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:40:23.497148 27363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:40:23.497189 27363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:40:23.501200 27363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:40:23.501271 27363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:40:23.501333 27363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:40:23.506613 27363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:40:23.507684 27363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:40:23.508677 27363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:40:23.518114 27371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:23.518114 27369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:40:23.518108 27368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:40:23.518373 27363 server_base.cc:1034] running on GCE node
I20260926 14:40:23.518815 27363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:40:23.519420 27363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:40:23.520583 27363 hybrid_clock.cc:648] HybridClock initialized: now 1790433623520555 us; error 41 us; skew 500 ppm
Sep 26 14:40:23 dist-test-slave-x4sn krb5kdc[26985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:40:23.523171 27363 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:40:23.524042 27363 webserver.cc:460] Webserver started at http://127.16.90.195:44027/ using document root <none> and password file <none>
I20260926 14:40:23.524570 27363 fs_manager.cc:359] Metadata directory not provided
I20260926 14:40:23.524636 27363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:40:23.524821 27363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:40:23.526808 27363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "11e8fb37a3c04a02b1207f4e8e9918e7"
format_stamp: "Formatted at 2026-09-26 14:40:23 on dist-test-slave-x4sn"
server_key: "852e44524390d98ca2286938c7b6401e"
server_key_iv: "79ef3661b46dd25f3c66deacd89a1b68"
server_key_version: "encryptionkey@0"
I20260926 14:40:23.527333 27363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "11e8fb37a3c04a02b1207f4e8e9918e7"
format_stamp: "Formatted at 2026-09-26 14:40:23 on dist-test-slave-x4sn"
server_key: "852e44524390d98ca2286938c7b6401e"
server_key_iv: "79ef3661b46dd25f3c66deacd89a1b68"
server_key_version: "encryptionkey@0"
I20260926 14:40:23.531159 27363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:40:23.533631 27378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:23.534816 27363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:40:23.534919 27363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "11e8fb37a3c04a02b1207f4e8e9918e7"
format_stamp: "Formatted at 2026-09-26 14:40:23 on dist-test-slave-x4sn"
server_key: "852e44524390d98ca2286938c7b6401e"
server_key_iv: "79ef3661b46dd25f3c66deacd89a1b68"
server_key_version: "encryptionkey@0"
I20260926 14:40:23.535006 27363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:40:23.558600 27363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:40:23.562242 27363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:40:23.562412 27363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:40:23.563129 27363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:40:23.564206 27363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:40:23.564270 27363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:23.564335 27363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:40:23.564368 27363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:40:23.573472 27363 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:37091
I20260926 14:40:23.573494 27491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:37091 every 8 connection(s)
I20260926 14:40:23.574318 27363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:40:23.574980 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 27363
I20260926 14:40:23.575086 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:40:23.575349 16747 external_mini_cluster.cc:1428] Setting key af046e7869baf3a688024312ed9c6a34
Sep 26 14:40:23 dist-test-slave-x4sn krb5kdc[26985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:40:23.587744 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.576923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:43533 (local address 127.16.90.254:35861)
0926 14:40:23.577042 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:23.577046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:23.578097 (+ 1051us) server_negotiation.cc:407] Connection header received
0926 14:40:23.579442 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:23.579445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:23.579493 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:23.579562 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:23.580688 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.581206 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.582275 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.582475 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.582545 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.584725 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:23.584728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:23.584730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:40:23.584756 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:23.586018 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:23.586461 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:23.586464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:23.586465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:23.586550 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:23.586819 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:23.586822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:23.586823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:23.586995 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:23.587067 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:23.587471 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 14:40:23.587590 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:40:23.588832 27494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.577462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35861 (local address 127.16.90.195:43533)
0926 14:40:23.577949 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:23.578002 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:23.579249 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:23.579730 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:23.579740 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:23.580087 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:40:23.580562 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.580574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.581300 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.581302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.582187 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.582196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.582578 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.582580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.582733 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.583225 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:40:23.583240 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:40:23.584580 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:40:23.586113 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:23.586117 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:23.586128 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:23.586356 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:23.586634 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:40:23.586636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:40:23.586638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:40:23.586757 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:40:23.587062 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:40:23.587067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:40:23.587286 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 14:40:23.587432 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 14:40:23.587786 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
I20260926 14:40:23.589818 27492 heartbeater.cc:347] Connected to a master server at 127.16.90.254:35861
I20260926 14:40:23.590039 27492 heartbeater.cc:464] Registering TS with master...
I20260926 14:40:23.590521 27492 heartbeater.cc:511] Master 127.16.90.254:35861 requested a full tablet report, sending...
I20260926 14:40:23.591497 27031 ts_manager.cc:194] Registered new tserver with Master: 11e8fb37a3c04a02b1207f4e8e9918e7 (127.16.90.195:37091)
I20260926 14:40:23.592082 27031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:43533
I20260926 14:40:23.600997 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:40:23.609326 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.602870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.16.90.254:35861)
0926 14:40:23.602992 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:23.602996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:23.603137 (+ 141us) server_negotiation.cc:407] Connection header received
0926 14:40:23.603246 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:23.603248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:23.603288 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:23.603348 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:40:23.604022 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.604533 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.605295 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.605509 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.605580 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.606843 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:40:23.606845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:40:23.606847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:40:23.606863 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:40:23.608000 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:23.608360 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:23.608362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:23.608363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:23.608428 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:40:23.608647 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:40:23.608648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:40:23.608650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:40:23.608852 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:40:23.608947 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:23.609099 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 14:40:23.609210 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:40:23.612030 27031 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:40:23.614960 27031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:40:23.629204 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.623254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:37091 (local address 127.0.0.1:38128)
0926 14:40:23.623901 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:23.623914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:23.623997 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:23.624940 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:23.624946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:23.624982 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:40:23.625150 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.625158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.626252 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.626256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.627566 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.627576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.628699 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.628702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.628789 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.628815 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:40:23.628880 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:40:23.629039 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":47,"spinlock_wait_cycles":6528,"thread_start_us":118,"threads_started":1}
I20260926 14:40:23.629377 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.623855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:41927 (local address 127.0.0.1:42742)
0926 14:40:23.624083 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:23.624097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:23.624169 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:23.625122 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:23.625124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:23.625139 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:40:23.625293 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.625300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.626645 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.626655 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.627834 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.627845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.629005 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.629009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.629095 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.629117 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:40:23.629177 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 14:40:23.629238 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":106,"thread_start_us":88,"threads_started":1}
I20260926 14:40:23.629591 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.623534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.16.90.195:37091)
0926 14:40:23.624507 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:23.624518 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:23.624541 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:40:23.624612 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:23.624617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:23.624809 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:23.624959 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:40:23.625272 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.626124 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.627710 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.628535 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.629165 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.629305 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:23.629352 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:40:23.629439 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":87,"threads_started":1}
I20260926 14:40:23.629767 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.624266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.16.90.194:41927)
0926 14:40:23.624643 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:23.624654 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:23.624672 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:40:23.624733 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:23.624737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:23.624879 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:23.624992 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:40:23.625411 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.626510 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.627967 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.628727 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.628813 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.628947 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:23.629532 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 14:40:23.629625 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":105,"threads_started":1}
I20260926 14:40:23.629765 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.623121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:46655 (local address 127.0.0.1:37464)
0926 14:40:23.623815 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:23.623866 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:23.623978 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:23.624940 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:23.624946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:23.624980 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:40:23.625174 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.625180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.626620 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.626624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.627843 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.627851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.628640 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.628644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.628759 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.628799 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 14:40:23.628900 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 14:40:23.629043 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":171,"spinlock_wait_cycles":5760,"thread_start_us":81,"threads_started":1}
I20260926 14:40:23.629194 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.623270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.16.90.193:46655)
0926 14:40:23.624279 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:23.624290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:23.624313 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:40:23.624386 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:23.624391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:23.624611 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:23.624765 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:40:23.625288 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.626491 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.627948 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.628486 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.628583 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.628791 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:23.628940 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 14:40:23.629039 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":116,"threads_started":1}
I20260926 14:40:23.632825 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 32184fa29c6a40eaaedbf0ff42148a89 (DEFAULT_TABLE table=test-table [id=e2c7d12c71fb489c99869bb32afcf725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:40:23.632681 27292 tablet_service.cc:1467] Processing CreateTablet for tablet 32184fa29c6a40eaaedbf0ff42148a89 (DEFAULT_TABLE table=test-table [id=e2c7d12c71fb489c99869bb32afcf725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:40:23.632671 27426 tablet_service.cc:1467] Processing CreateTablet for tablet 32184fa29c6a40eaaedbf0ff42148a89 (DEFAULT_TABLE table=test-table [id=e2c7d12c71fb489c99869bb32afcf725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:40:23.633754 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32184fa29c6a40eaaedbf0ff42148a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:23.633760 27426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32184fa29c6a40eaaedbf0ff42148a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:23.633756 27292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32184fa29c6a40eaaedbf0ff42148a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:40:23.640232 27510 tablet_bootstrap.cc:492] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40: Bootstrap starting.
I20260926 14:40:23.641173 27511 tablet_bootstrap.cc:492] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd: Bootstrap starting.
I20260926 14:40:23.642359 27510 tablet_bootstrap.cc:654] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:23.643221 27511 tablet_bootstrap.cc:654] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:23.643374 27510 log.cc:824] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40: Log is configured to *not* fsync() on all Append() calls
I20260926 14:40:23.643986 27512 tablet_bootstrap.cc:492] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7: Bootstrap starting.
I20260926 14:40:23.645336 27510 tablet_bootstrap.cc:492] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40: No bootstrap required, opened a new log
I20260926 14:40:23.645507 27510 ts_tablet_manager.cc:1397] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260926 14:40:23.645768 27512 tablet_bootstrap.cc:654] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7: Neither blocks nor log segments found. Creating new log.
I20260926 14:40:23.646708 27512 log.cc:824] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7: Log is configured to *not* fsync() on all Append() calls
I20260926 14:40:23.647344 27511 log.cc:824] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd: Log is configured to *not* fsync() on all Append() calls
I20260926 14:40:23.648686 27512 tablet_bootstrap.cc:492] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7: No bootstrap required, opened a new log
I20260926 14:40:23.648649 27510 raft_consensus.cc:348] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46655 } } peers { permanent_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 37091 } } peers { permanent_uuid: "bc7670bc20254a71b6c1ec14e83f58cd" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41927 } }
I20260926 14:40:23.648866 27512 ts_tablet_manager.cc:1397] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:40:23.648916 27510 raft_consensus.cc:374] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:23.648957 27511 tablet_bootstrap.cc:492] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd: No bootstrap required, opened a new log
I20260926 14:40:23.648968 27510 raft_consensus.cc:729] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9c144e14bc4aeebe9cf1ba2aec2d40, State: Initialized, Role: FOLLOWER
I20260926 14:40:23.649121 27511 ts_tablet_manager.cc:1397] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260926 14:40:23.649425 27510 consensus_queue.cc:260] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46655 } } peers { permanent_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 37091 } } peers { permanent_uuid: "bc7670bc20254a71b6c1ec14e83f58cd" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41927 } }
I20260926 14:40:23.650394 27510 ts_tablet_manager.cc:1428] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260926 14:40:23.650745 27223 heartbeater.cc:503] Master 127.16.90.254:35861 was elected leader, sending a full tablet report...
I20260926 14:40:23.652150 27512 raft_consensus.cc:348] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46655 } } peers { permanent_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 37091 } } peers { permanent_uuid: "bc7670bc20254a71b6c1ec14e83f58cd" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41927 } }
I20260926 14:40:23.652359 27512 raft_consensus.cc:374] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:23.652408 27512 raft_consensus.cc:729] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e8fb37a3c04a02b1207f4e8e9918e7, State: Initialized, Role: FOLLOWER
I20260926 14:40:23.652632 27511 raft_consensus.cc:348] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46655 } } peers { permanent_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 37091 } } peers { permanent_uuid: "bc7670bc20254a71b6c1ec14e83f58cd" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41927 } }
I20260926 14:40:23.652788 27512 consensus_queue.cc:260] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46655 } } peers { permanent_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 37091 } } peers { permanent_uuid: "bc7670bc20254a71b6c1ec14e83f58cd" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41927 } }
I20260926 14:40:23.652884 27511 raft_consensus.cc:374] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:40:23.652942 27511 raft_consensus.cc:729] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7670bc20254a71b6c1ec14e83f58cd, State: Initialized, Role: FOLLOWER
I20260926 14:40:23.653342 27511 consensus_queue.cc:260] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46655 } } peers { permanent_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 37091 } } peers { permanent_uuid: "bc7670bc20254a71b6c1ec14e83f58cd" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41927 } }
I20260926 14:40:23.653565 27512 ts_tablet_manager.cc:1428] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:40:23.653903 27492 heartbeater.cc:503] Master 127.16.90.254:35861 was elected leader, sending a full tablet report...
I20260926 14:40:23.654249 27511 ts_tablet_manager.cc:1428] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260926 14:40:23.654546 27358 heartbeater.cc:503] Master 127.16.90.254:35861 was elected leader, sending a full tablet report...
W20260926 14:40:23.655748 27224 tablet.cc:2369] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:40:23.825248 27493 tablet.cc:2369] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:40:23.839205 27516 raft_consensus.cc:482] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:40:23.839416 27516 raft_consensus.cc:504] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46655 } } peers { permanent_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 37091 } } peers { permanent_uuid: "bc7670bc20254a71b6c1ec14e83f58cd" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41927 } }
I20260926 14:40:23.840595 27516 leader_election.cc:288] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11e8fb37a3c04a02b1207f4e8e9918e7 (127.16.90.195:37091), bc7670bc20254a71b6c1ec14e83f58cd (127.16.90.194:41927)
I20260926 14:40:23.844805 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.841397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:42385 (local address 127.16.90.194:41927)
0926 14:40:23.841538 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:23.841543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:23.841726 (+ 183us) server_negotiation.cc:407] Connection header received
0926 14:40:23.841823 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:23.841825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:23.841867 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:23.841923 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:40:23.842203 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.842762 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.843824 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.844313 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.844422 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.844459 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:23.844613 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 14:40:23.844676 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:40:23.844851 27520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.841229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:41927 (local address 127.16.90.193:42385)
0926 14:40:23.841627 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:23.841639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:23.841738 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:23.841948 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:23.841951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:23.841968 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:40:23.842088 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.842094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.842873 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.842876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.843719 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.843729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.844407 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.844411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.844503 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.844527 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:40:23.844603 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 14:40:23.844682 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":19,"thread_start_us":124,"threads_started":1}
I20260926 14:40:23.844864 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.840963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:59581 (local address 127.16.90.195:37091)
0926 14:40:23.841084 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:23.841088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:23.841199 (+ 111us) server_negotiation.cc:407] Connection header received
0926 14:40:23.841347 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:23.841350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:23.841402 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:23.841492 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:40:23.841804 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.842444 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.843633 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.844136 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.844239 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.844290 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:23.844695 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 14:40:23.844766 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:40:23.845023 27519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.840854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:37091 (local address 127.16.90.193:59581)
0926 14:40:23.841105 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:23.841121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:40:23.841240 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:40:23.841525 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:40:23.841528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:40:23.841551 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:40:23.841671 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.841677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.842556 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.842560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.843514 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:40:23.843524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.844472 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.844478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:40:23.844569 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.844588 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:40:23.844651 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:40:23.844702 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":25,"thread_start_us":86,"threads_started":1}
I20260926 14:40:23.845662 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32184fa29c6a40eaaedbf0ff42148a89" candidate_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7" is_pre_election: true
I20260926 14:40:23.845660 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32184fa29c6a40eaaedbf0ff42148a89" candidate_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc7670bc20254a71b6c1ec14e83f58cd" is_pre_election: true
I20260926 14:40:23.845952 27312 raft_consensus.cc:2454] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b9c144e14bc4aeebe9cf1ba2aec2d40 in term 0.
I20260926 14:40:23.845949 27446 raft_consensus.cc:2454] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b9c144e14bc4aeebe9cf1ba2aec2d40 in term 0.
I20260926 14:40:23.846396 27110 leader_election.cc:302] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11e8fb37a3c04a02b1207f4e8e9918e7, 2b9c144e14bc4aeebe9cf1ba2aec2d40; no voters:
I20260926 14:40:23.846647 27516 raft_consensus.cc:2781] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:40:23.846736 27516 raft_consensus.cc:482] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:40:23.846784 27516 raft_consensus.cc:3037] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:23.847805 27516 raft_consensus.cc:504] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46655 } } peers { permanent_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 37091 } } peers { permanent_uuid: "bc7670bc20254a71b6c1ec14e83f58cd" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41927 } }
I20260926 14:40:23.848110 27516 leader_election.cc:288] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [CANDIDATE]: Term 1 election: Requested vote from peers 11e8fb37a3c04a02b1207f4e8e9918e7 (127.16.90.195:37091), bc7670bc20254a71b6c1ec14e83f58cd (127.16.90.194:41927)
I20260926 14:40:23.848439 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32184fa29c6a40eaaedbf0ff42148a89" candidate_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7"
I20260926 14:40:23.848486 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32184fa29c6a40eaaedbf0ff42148a89" candidate_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc7670bc20254a71b6c1ec14e83f58cd"
I20260926 14:40:23.848547 27446 raft_consensus.cc:3037] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:23.848592 27312 raft_consensus.cc:3037] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:40:23.849541 27446 raft_consensus.cc:2454] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b9c144e14bc4aeebe9cf1ba2aec2d40 in term 1.
I20260926 14:40:23.849598 27312 raft_consensus.cc:2454] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b9c144e14bc4aeebe9cf1ba2aec2d40 in term 1.
I20260926 14:40:23.849809 27110 leader_election.cc:302] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11e8fb37a3c04a02b1207f4e8e9918e7, 2b9c144e14bc4aeebe9cf1ba2aec2d40; no voters:
I20260926 14:40:23.849970 27516 raft_consensus.cc:2781] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:40:23.850199 27516 raft_consensus.cc:686] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [term 1 LEADER]: Becoming Leader. State: Replica: 2b9c144e14bc4aeebe9cf1ba2aec2d40, State: Running, Role: LEADER
I20260926 14:40:23.850514 27516 consensus_queue.cc:237] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46655 } } peers { permanent_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 37091 } } peers { permanent_uuid: "bc7670bc20254a71b6c1ec14e83f58cd" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41927 } }
I20260926 14:40:23.853650 27030 catalog_manager.cc:5495] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b9c144e14bc4aeebe9cf1ba2aec2d40 (127.16.90.193). New cstate: current_term: 1 leader_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9c144e14bc4aeebe9cf1ba2aec2d40" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc7670bc20254a71b6c1ec14e83f58cd" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41927 } health_report { overall_health: UNKNOWN } } }
W20260926 14:40:23.856703 27359 tablet.cc:2369] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:40:23.867514 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.864225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.16.90.193:46655)
0926 14:40:23.864352 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:40:23.864355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:40:23.864370 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:40:23.864499 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:40:23.864501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:40:23.864551 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:40:23.864634 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:40:23.864807 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.865301 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.866150 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:40:23.866386 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:40:23.866465 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:40:23.866700 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:40:23.867123 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:40:23.867205 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:40:23.867282 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 14:40:23.867340 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:40:23.873859 27446 raft_consensus.cc:1261] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7 [term 1 FOLLOWER]: Refusing update from remote peer 2b9c144e14bc4aeebe9cf1ba2aec2d40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:40:23.873859 27312 raft_consensus.cc:1261] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd [term 1 FOLLOWER]: Refusing update from remote peer 2b9c144e14bc4aeebe9cf1ba2aec2d40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:40:23.874532 27521 consensus_queue.cc:1035] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11e8fb37a3c04a02b1207f4e8e9918e7" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:40:23.874709 27516 consensus_queue.cc:1035] T 32184fa29c6a40eaaedbf0ff42148a89 P 2b9c144e14bc4aeebe9cf1ba2aec2d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc7670bc20254a71b6c1ec14e83f58cd" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:40:23.889879 27031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:40:23.890038 27031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:40:23.892370 27031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f23f3bf76727446e8db9e0b30c916847: Sending DeleteTablet for 3 replicas of tablet 32184fa29c6a40eaaedbf0ff42148a89
I20260926 14:40:23.893329 27292 tablet_service.cc:1514] Processing DeleteTablet for tablet 32184fa29c6a40eaaedbf0ff42148a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:23 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:42742
I20260926 14:40:23.893344 27426 tablet_service.cc:1514] Processing DeleteTablet for tablet 32184fa29c6a40eaaedbf0ff42148a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:40:23 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:38128
I20260926 14:40:23.893945 27535 tablet_replica.cc:330] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7: stopping tablet replica
I20260926 14:40:23.894255 27535 raft_consensus.cc:2229] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:40:23.894551 27535 raft_consensus.cc:2258] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:40:23.894662 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 27093
I20260926 14:40:23.898047 27535 ts_tablet_manager.cc:1905] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:40:23.900756 27535 ts_tablet_manager.cc:1918] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:40:23.900851 27535 log.cc:1196] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/32184fa29c6a40eaaedbf0ff42148a89
I20260926 14:40:23.901141 27534 tablet_replica.cc:330] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd: stopping tablet replica
I20260926 14:40:23.901170 27535 ts_tablet_manager.cc:1939] T 32184fa29c6a40eaaedbf0ff42148a89 P 11e8fb37a3c04a02b1207f4e8e9918e7: Deleting consensus metadata
I20260926 14:40:23.901535 27534 raft_consensus.cc:2229] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:40:23.901914 27534 raft_consensus.cc:2258] T 32184fa29c6a40eaaedbf0ff42148a89 P bc7670bc20254a71b6c1ec14e83f58cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:40:23.902493 27016 catalog_manager.cc:4841] TS 11e8fb37a3c04a02b1207f4e8e9918e7 (127.16.90.195:37091): tablet 32184fa29c6a40eaaedbf0ff42148a89 (table test-table [id=e2c7d12c71fb489c99869bb32afcf725]) successfully deleted
W20260926 14:40:23.902953 27018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:46655 (error 108)
I20260926 14:40:23.903208 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 27228
I20260926 14:40:23.904289 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.903999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:46655 (local address 127.0.0.1:37482)
0926 14:40:23.904131 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:23.904207 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 14:40:23.904989 27018 catalog_manager.cc:4584] TS 2b9c144e14bc4aeebe9cf1ba2aec2d40 (127.16.90.193:46655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32184fa29c6a40eaaedbf0ff42148a89: Network error: Client connection negotiation failed: client connection to 127.16.90.193:46655: connect: Connection refused (error 111)
I20260926 14:40:23.908967 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 27363
I20260926 14:40:23.910308 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:40:23.909074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:41927 (local address 127.0.0.1:42758)
0926 14:40:23.910168 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0926 14:40:23.910227 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.194:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1008}
I20260926 14:40:23.913709 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 26999
2026-09-26T14:40:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2274 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:40:23 dist-test-slave-x4sn krb5kdc[27539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:40:23 dist-test-slave-x4sn krb5kdc[27539](info): set up 2 sockets
Sep 26 14:40:23 dist-test-slave-x4sn krb5kdc[27539](info): commencing operation
krb5kdc: starting...
W20260926 14:40:24.894029 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:40:24 dist-test-slave-x4sn krb5kdc[27539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:40:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:40:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.90.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.90.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/rangeradmin_127.16.90.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/rangeradmin_127.16.90.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.90.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.90.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/rangerlookup_127.16.90.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/rangerlookup_127.16.90.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.90.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.90.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/HTTP_127.16.90.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/HTTP_127.16.90.212@KRBTEST.COM.keytab.
I20260926 14:40:25.060648 16747 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasks9vtzz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/postgres -l logfile start
2026-09-26 14:40:27.410 UTC [27569] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 14:40:27.410 UTC [27569] LOG: listening on IPv4 address "127.16.90.212", port 45757
2026-09-26 14:40:27.414 UTC [27569] LOG: listening on Unix socket "/tmp/.s.PGSQL.45757"
2026-09-26 14:40:27.434 UTC [27572] LOG: database system was shut down at 2026-09-26 14:40:26 UTC
2026-09-26 14:40:27.437 UTC [27569] LOG: database system is ready to accept connections
I20260926 14:40:28.313594 16747 mini_postgres.cc:96] Postgres bound to 45757
2026-09-26 14:40:28.319 UTC [27580] FATAL: database "slave" does not exist
127.16.90.212:45757 - accepting connections
I20260926 14:40:28.320343 16747 mini_ranger.cc:161] Starting Ranger...
I20260926 14:40:28.334703 16747 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 14:40:28.797089 16747 mini_ranger.cc:87] Created ranger Postgres database
I20260926 14:40:28.797179 16747 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-admin
2026-09-26 14:40:29,124 [I] DB FLAVOR :POSTGRES
2026-09-26 14:40:29,124 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:40:29,124 [I] Checking connection..
2026-09-26 14:40:29,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 14:40:29,393 [I] Checking connection passed.
2026-09-26 14:40:29,394 [I] --------- Verifying version history table ---------
2026-09-26 14:40:29,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:40:29,650 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:40:29,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:40:29,906 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:40:29,907 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 14:40:29,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 14:40:30.159 UTC [27644] WARNING: there is no transaction in progress
2026-09-26 14:40:30,166 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 14:40:30,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:40:30,422 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 14:40:30,423 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:40:30,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 14:40:30,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 14:40:30,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-x4sn.c.gcp-upstream.internal','N') ;"
2026-09-26 14:40:31,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:40:31,387 [I] Table x_portal_user does not exist in database ranger
2026-09-26 14:40:31,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:40:31,623 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 14:40:31,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:40:31,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:40:32,095 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 14:40:32,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 14:40:32.994 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.010 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.026 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.040 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.055 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.072 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.123 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.130 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.143 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.153 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.162 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.176 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.185 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.191 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.197 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33.203 UTC [27752] WARNING: there is no transaction in progress
2026-09-26 14:40:33,799 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 14:40:33,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 14:40:34,029 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 14:40:34,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:40:34,274 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 14:40:34,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:40:34,511 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 14:40:34,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:40:34,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:40:34,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-x4sn.c.gcp-upstream.internal';"
2026-09-26 14:40:35,220 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 14:40:35,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 14:40:35,470 [I] DB_PATCHES have already been applied
I20260926 14:40:35.476284 16747 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-admin:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasks9vtzz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 14:40:35.476356 16747 mini_ranger.cc:193] Using host: 127.16.90.212
I20260926 14:40:35.479811 16747 mini_ranger.cc:227] Ranger admin URL: http://127.16.90.212:43223
Sep 26, 2026 2:40:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:40:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:40:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:40:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:40:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:40:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:40:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 14:40:36.459038 16747 mini_ranger.cc:160] Time spent starting Ranger: real 8.139s user 0.000s sys 0.010s
Sep 26, 2026 2:40:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:40:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 2:40:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:40:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:40:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:40:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 14:40:56.602978 16747 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.90.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.90.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/rangerkms_127.16.90.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/rangerkms_127.16.90.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.90.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.90.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.90.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/HTTP_127.16.90.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.90.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/HTTP_127.16.90.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 14:40:56 dist-test-slave-x4sn krb5kdc[27539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433656, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 14:40:56.698139 16747 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 14:40:56.707590 16747 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 14:40:57.512975 16747 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 14:40:57.513054 16747 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-kms
I20260926 14:40:57.513131 16747 mini_ranger_kms.cc:227] Using postgres at 127.16.90.212:45757
2026-09-26 14:40:57,719 [I] DB FLAVOR :POSTGRES
2026-09-26 14:40:57,719 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:40:57,719 [I] Checking connection
2026-09-26 14:40:57,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 14:40:57,979 [I] connection success
2026-09-26 14:40:57,979 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 14:40:57,980 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 14:40:57,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 14:40:58,220 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 14:40:58,220 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:40:58,220 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 14:40:58,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:45757/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 14:40:58,490 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 14:40:58.931797 16747 mini_ranger_kms.cc:310] Created kms service
I20260926 14:40:59.125176 16747 mini_ranger_kms.cc:326] Created kudu user
I20260926 14:40:59.165441 16747 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 14:40:59 dist-test-slave-x4sn krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433656, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:40:59.550370 16747 mini_ranger_kms.cc:384] Added ranger policy
I20260926 14:40:59.550520 16747 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-kms:/tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 14:40:59.550542 16747 mini_ranger_kms.cc:240] Using host: 127.16.90.212
I20260926 14:40:59.554286 16747 mini_ranger_kms.cc:276] Ranger KMS PID: 27933
I20260926 14:40:59.554425 16747 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.90.212:34383
Sep 26, 2026 2:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 14:41:01.553049 16747 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.855s user 0.008s sys 0.182s
I20260926 14:41:01.553184 16747 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 14:41:01.553292 16747 mini_ranger_kms.cc:401] 127.16.90.212:34383/kms/v1/keys
Sep 26 14:41:01 dist-test-slave-x4sn krb5kdc[27539](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790433661, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.90.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 14:41:02 dist-test-slave-x4sn krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433661, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.90.212@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
Sep 26 14:41:02 dist-test-slave-x4sn krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433661, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.90.212@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
Sep 26 14:41:04 dist-test-slave-x4sn krb5kdc[27539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:41:04.831225 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:32921
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:33719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:32921
--ranger_config_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.90.212:34383/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:41:04.938634 27984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:04.938879 27984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:04.938932 27984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:04.942893 27984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:41:04.942953 27984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:04.942978 27984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:04.942998 27984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:41:04.943018 27984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:41:04.948290 27984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:33719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.90.212:34383/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:32921
--ranger_config_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:32921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:04.949260 27984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:04.950215 27984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:04.958276 27989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:04.958269 27990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:04.958268 27992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:04.958405 27984 server_base.cc:1034] running on GCE node
I20260926 14:41:04.959049 27984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:04.960109 27984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:04.961289 27984 hybrid_clock.cc:648] HybridClock initialized: now 1790433664961264 us; error 35 us; skew 500 ppm
Sep 26 14:41:04 dist-test-slave-x4sn krb5kdc[27539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433664, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:04.963917 27984 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:41:04.964743 27984 webserver.cc:460] Webserver started at http://127.16.90.254:44129/ using document root <none> and password file <none>
I20260926 14:41:04.965234 27984 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:04.965294 27984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:41:04.965502 27984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:41:04 dist-test-slave-x4sn krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433664, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:05.022435 27984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "9c7d546817264800b1d8b618b0150b38"
format_stamp: "Formatted at 2026-09-26 14:41:05 on dist-test-slave-x4sn"
server_key: "bd4c71c1d89fa6f033688db32f343545"
server_key_iv: "1c7e4c7a98765e9f7f83c080e6e56617"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:05.023104 27984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "9c7d546817264800b1d8b618b0150b38"
format_stamp: "Formatted at 2026-09-26 14:41:05 on dist-test-slave-x4sn"
server_key: "bd4c71c1d89fa6f033688db32f343545"
server_key_iv: "1c7e4c7a98765e9f7f83c080e6e56617"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:05.027220 27984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:41:05.041863 27999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:05.043267 27984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:41:05.043395 27984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "9c7d546817264800b1d8b618b0150b38"
format_stamp: "Formatted at 2026-09-26 14:41:05 on dist-test-slave-x4sn"
server_key: "bd4c71c1d89fa6f033688db32f343545"
server_key_iv: "1c7e4c7a98765e9f7f83c080e6e56617"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:05.043520 27984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:05.064746 27984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:05.067540 27984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:05.067759 27984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:05.075412 27984 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:32921
I20260926 14:41:05.075450 28052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:32921 every 8 connection(s)
I20260926 14:41:05.076396 27984 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:41:05.078373 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 27984
I20260926 14:41:05.078502 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:41:05.078799 16747 external_mini_cluster.cc:1428] Setting key 97665bebf2b58cda1942a799051e1f6f
I20260926 14:41:05.079361 28053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:41:05.086161 28053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38: Bootstrap starting.
I20260926 14:41:05.088883 28053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38: Neither blocks nor log segments found. Creating new log.
Sep 26 14:41:05 dist-test-slave-x4sn krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:05.089617 28053 log.cc:824] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38: Log is configured to *not* fsync() on all Append() calls
I20260926 14:41:05.091861 28053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38: No bootstrap required, opened a new log
I20260926 14:41:05.095212 28056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:05.080306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37616 (local address 127.16.90.254:32921)
0926 14:41:05.080766 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:05.080785 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:05.080830 (+ 45us) server_negotiation.cc:407] Connection header received
0926 14:41:05.081734 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:05.081757 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:05.082192 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:05.082583 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:05.083415 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:05.085020 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:05.086182 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:05.086890 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:05.087044 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:05.090073 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:05.090080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:05.090096 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:05.090123 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:05.092393 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:05.092855 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:05.092860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:05.092864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:05.092982 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:05.093267 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:05.093270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:05.093272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:05.093672 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:05.093877 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:05.094186 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 14:41:05.094490 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":145,"threads_started":1}
I20260926 14:41:05.095875 28053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c7d546817264800b1d8b618b0150b38" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 32921 } }
I20260926 14:41:05.096164 28053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:41:05.096264 28053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c7d546817264800b1d8b618b0150b38, State: Initialized, Role: FOLLOWER
I20260926 14:41:05.096732 28053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c7d546817264800b1d8b618b0150b38" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 32921 } }
I20260926 14:41:05.096886 28053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:41:05.096971 28053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:41:05.097067 28053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:41:05.098179 28053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c7d546817264800b1d8b618b0150b38" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 32921 } }
I20260926 14:41:05.098523 28053 leader_election.cc:302] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c7d546817264800b1d8b618b0150b38; no voters:
I20260926 14:41:05.098853 28053 leader_election.cc:288] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:41:05.098943 28058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:41:05.099215 28058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [term 1 LEADER]: Becoming Leader. State: Replica: 9c7d546817264800b1d8b618b0150b38, State: Running, Role: LEADER
I20260926 14:41:05.099555 28058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c7d546817264800b1d8b618b0150b38" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 32921 } }
I20260926 14:41:05.100148 28053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:41:05.101039 28059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c7d546817264800b1d8b618b0150b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c7d546817264800b1d8b618b0150b38" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 32921 } } }
I20260926 14:41:05.101229 28059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [sys.catalog]: This master's current role is: LEADER
I20260926 14:41:05.101504 28059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c7d546817264800b1d8b618b0150b38. Latest consensus state: current_term: 1 leader_uuid: "9c7d546817264800b1d8b618b0150b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c7d546817264800b1d8b618b0150b38" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 32921 } } }
I20260926 14:41:05.101636 28059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38 [sys.catalog]: This master's current role is: LEADER
I20260926 14:41:05.101955 28067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:41:05.102320 28053 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 14:41:05.108316 28067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:41:05.119859 28067 catalog_manager.cc:1347] Generated new cluster ID: 6019ccd4fb664d8480c986a5bffca35b
I20260926 14:41:05.119973 28067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:41:05.147087 28067 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:41:05.148676 28067 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:41:05.157652 28067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38: Generated new TSK 0
I20260926 14:41:05.158543 28067 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:41:05.807075 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.90.212:34383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:32921
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:33719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:41:05.928609 28098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:05.928892 28098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:05.928951 28098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:05.933050 28098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:05.933115 28098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:05.933194 28098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:41:05.938642 28098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:33719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.90.212:34383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:32921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:05.939714 28098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:05.940732 28098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:05.950435 28103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:05.950434 28104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:05.950666 28106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:05.950780 28098 server_base.cc:1034] running on GCE node
I20260926 14:41:05.951252 28098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:05.951937 28098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:05.953112 28098 hybrid_clock.cc:648] HybridClock initialized: now 1790433665953085 us; error 42 us; skew 500 ppm
Sep 26 14:41:05 dist-test-slave-x4sn krb5kdc[27539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:05.956028 28098 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:41:05.956831 28098 webserver.cc:460] Webserver started at http://127.16.90.193:33895/ using document root <none> and password file <none>
I20260926 14:41:05.957376 28098 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:05.957441 28098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:41:05.957655 28098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:41:05 dist-test-slave-x4sn krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:05.973920 28098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "1872b4ac0b024d57801fad6520ab3f2c"
format_stamp: "Formatted at 2026-09-26 14:41:05 on dist-test-slave-x4sn"
server_key: "aeddf58c9b0e025a267bdcbf1e97344a"
server_key_iv: "328eb8510ab15e6e337f88cda9e1b6ab"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:05.974640 28098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "1872b4ac0b024d57801fad6520ab3f2c"
format_stamp: "Formatted at 2026-09-26 14:41:05 on dist-test-slave-x4sn"
server_key: "aeddf58c9b0e025a267bdcbf1e97344a"
server_key_iv: "328eb8510ab15e6e337f88cda9e1b6ab"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:05.979591 28098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 14:41:05.998400 28115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:06.000147 28098 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 14:41:06.000306 28098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "1872b4ac0b024d57801fad6520ab3f2c"
format_stamp: "Formatted at 2026-09-26 14:41:05 on dist-test-slave-x4sn"
server_key: "aeddf58c9b0e025a267bdcbf1e97344a"
server_key_iv: "328eb8510ab15e6e337f88cda9e1b6ab"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:06.000444 28098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:06.027690 28098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:06.031302 28098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:06.031558 28098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:06.032347 28098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:41:06.033442 28098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:41:06.033550 28098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:06.033689 28098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:41:06.033818 28098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:06.048295 28098 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:37369
I20260926 14:41:06.049206 28229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:37369 every 8 connection(s)
I20260926 14:41:06.049603 28098 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:41:06.054649 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 28098
I20260926 14:41:06.054793 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:41:06.055119 16747 external_mini_cluster.cc:1428] Setting key 84f7dfa6b12428700c51f69534bd1e60
Sep 26 14:41:06 dist-test-slave-x4sn krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:06.072597 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.052460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:32921 (local address 127.16.90.193:37641)
0926 14:41:06.053093 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:06.053153 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:06.054193 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:06.055881 (+ 1688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:06.055891 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:06.056303 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:06.059056 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.059078 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.060310 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.060314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.061728 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.061739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.062514 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.062518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.062729 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.065330 (+ 2601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:06.065353 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:06.067070 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:06.069253 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:06.069260 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:06.069277 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:06.069602 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:06.069925 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:06.069927 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:06.069929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:06.070114 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:06.070595 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:06.070602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:06.070937 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 14:41:06.071149 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 14:41:06.071499 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":172,"threads_started":1}
I20260926 14:41:06.073366 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.053658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:37641 (local address 127.16.90.254:32921)
0926 14:41:06.053952 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:06.053964 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:06.053985 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:41:06.054444 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:06.054448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:06.054506 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:06.054588 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:06.059241 (+ 4653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.060169 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.061870 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.062235 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.062333 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.067271 (+ 4938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:06.067275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:06.067278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:06.067304 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:06.069121 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:06.069731 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:06.069735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:06.069737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:06.069828 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:06.070224 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:06.070227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:06.070229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:06.070452 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:06.070543 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:06.073068 (+ 2525us) server_negotiation.cc:299] Negotiation successful
0926 14:41:06.073198 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":109,"threads_started":1}
I20260926 14:41:06.075145 28230 heartbeater.cc:347] Connected to a master server at 127.16.90.254:32921
I20260926 14:41:06.075405 28230 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:06.075963 28230 heartbeater.cc:511] Master 127.16.90.254:32921 requested a full tablet report, sending...
I20260926 14:41:06.077627 28017 ts_manager.cc:194] Registered new tserver with Master: 1872b4ac0b024d57801fad6520ab3f2c (127.16.90.193:37369)
I20260926 14:41:06.079044 28017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:37641
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:41:06.136372 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.90.212:34383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:32921
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:33719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:41:06 dist-test-slave-x4sn krb5kdc[27539](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790433666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 14:41:06.303886 28238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:06.304396 28238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:06.304525 28238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:06.310585 28238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:06.310722 28238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:06.310860 28238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:41:06.318579 28238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:33719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.90.212:34383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:32921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:06.320303 28238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:06.321753 28238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:06.339464 28245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:06.339471 28244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:06.340512 28238 server_base.cc:1034] running on GCE node
W20260926 14:41:06.343778 28247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:06.344381 28238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:06.345234 28238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:06.346560 28238 hybrid_clock.cc:648] HybridClock initialized: now 1790433666346492 us; error 88 us; skew 500 ppm
Sep 26 14:41:06 dist-test-slave-x4sn krb5kdc[27539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:06.350277 28238 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:41:06.351655 28238 webserver.cc:460] Webserver started at http://127.16.90.194:44027/ using document root <none> and password file <none>
I20260926 14:41:06.352423 28238 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:06.352512 28238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:41:06.352756 28238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:41:06 dist-test-slave-x4sn krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:06.369343 28238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "31efa6b771ee4931a9c528205b26850f"
format_stamp: "Formatted at 2026-09-26 14:41:06 on dist-test-slave-x4sn"
server_key: "c9e4c34f93b6f8329f8c2ece15e9e19b"
server_key_iv: "880e841b6dd0176ad73f5c094cf16a59"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:06.370044 28238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "31efa6b771ee4931a9c528205b26850f"
format_stamp: "Formatted at 2026-09-26 14:41:06 on dist-test-slave-x4sn"
server_key: "c9e4c34f93b6f8329f8c2ece15e9e19b"
server_key_iv: "880e841b6dd0176ad73f5c094cf16a59"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:06.374743 28238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 14:41:06.397195 28254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:06.399462 28238 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:41:06.399600 28238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "31efa6b771ee4931a9c528205b26850f"
format_stamp: "Formatted at 2026-09-26 14:41:06 on dist-test-slave-x4sn"
server_key: "c9e4c34f93b6f8329f8c2ece15e9e19b"
server_key_iv: "880e841b6dd0176ad73f5c094cf16a59"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:06.399783 28238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:06.414455 28238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:06.418108 28238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:06.418361 28238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:06.419077 28238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:41:06.420197 28238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:41:06.420302 28238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:06.420399 28238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:41:06.420462 28238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:06.433976 28238 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:41293
I20260926 14:41:06.435256 28238 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:41:06.443001 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 28238
I20260926 14:41:06.443140 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:41:06.443450 16747 external_mini_cluster.cc:1428] Setting key e3cee965b99cd218b5a604e43fc3cbb1
I20260926 14:41:06.436678 28367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:41293 every 8 connection(s)
Sep 26 14:41:06 dist-test-slave-x4sn krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:06.459317 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.440357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:49515 (local address 127.16.90.254:32921)
0926 14:41:06.440474 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:06.440478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:06.440507 (+ 29us) server_negotiation.cc:407] Connection header received
0926 14:41:06.446239 (+ 5732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:06.446243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:06.446293 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:06.446342 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:06.448344 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.449175 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.451276 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.451624 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.451736 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.454751 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:06.454754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:06.454757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:06.454784 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:06.456540 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:06.457143 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:06.457147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:06.457148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:06.457233 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:06.457610 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:06.457613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:06.457615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:06.457824 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:06.457893 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:06.458951 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0926 14:41:06.459134 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:41:06.460312 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.439326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:32921 (local address 127.16.90.194:49515)
0926 14:41:06.439843 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:06.439896 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:06.443412 (+ 3516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:06.446818 (+ 3406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:06.446828 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:06.447316 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:06.448167 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.448189 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.449334 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.449337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.451144 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.451155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.451940 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.451944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.452151 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.452792 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:06.452812 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:06.454562 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:06.456673 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:06.456679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:06.456694 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:06.457027 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:06.457333 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:06.457336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:06.457338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:06.457505 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:06.458369 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:06.458376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:06.458685 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 14:41:06.458899 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 14:41:06.459238 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":124,"threads_started":1}
I20260926 14:41:06.461743 28368 heartbeater.cc:347] Connected to a master server at 127.16.90.254:32921
I20260926 14:41:06.462001 28368 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:06.462584 28368 heartbeater.cc:511] Master 127.16.90.254:32921 requested a full tablet report, sending...
I20260926 14:41:06.463828 28017 ts_manager.cc:194] Registered new tserver with Master: 31efa6b771ee4931a9c528205b26850f (127.16.90.194:41293)
I20260926 14:41:06.464605 28017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:49515
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:41:06.488994 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.90.212:34383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:32921
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:33719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:41:06.664885 28373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:06.665383 28373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:06.665519 28373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:06.671523 28373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:06.671712 28373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:06.671870 28373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:41:06.679649 28373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:33719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.90.212:34383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:32921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:06.681336 28373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:06.682760 28373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:06.705996 28379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:06.706349 28381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:06.707149 28378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:06.707886 28373 server_base.cc:1034] running on GCE node
I20260926 14:41:06.708388 28373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:06.709165 28373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:06.710419 28373 hybrid_clock.cc:648] HybridClock initialized: now 1790433666710351 us; error 86 us; skew 500 ppm
Sep 26 14:41:06 dist-test-slave-x4sn krb5kdc[27539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:06.713876 28373 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:41:06.714912 28373 webserver.cc:460] Webserver started at http://127.16.90.195:37507/ using document root <none> and password file <none>
I20260926 14:41:06.715574 28373 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:06.715637 28373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:41:06.715870 28373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:41:06 dist-test-slave-x4sn krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:06.748018 28373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "0c89fe97418149f29f65f9a80fad094f"
format_stamp: "Formatted at 2026-09-26 14:41:06 on dist-test-slave-x4sn"
server_key: "cd0844174f3716baa0befd478f8744c2"
server_key_iv: "98132cf5b49384b32f658e630ac63a6d"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:06.749773 28373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "0c89fe97418149f29f65f9a80fad094f"
format_stamp: "Formatted at 2026-09-26 14:41:06 on dist-test-slave-x4sn"
server_key: "cd0844174f3716baa0befd478f8744c2"
server_key_iv: "98132cf5b49384b32f658e630ac63a6d"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:06.758985 28373 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260926 14:41:06.781661 28388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Sep 26 14:41:06 dist-test-slave-x4sn krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:06.783438 28373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 14:41:06.783625 28373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "0c89fe97418149f29f65f9a80fad094f"
format_stamp: "Formatted at 2026-09-26 14:41:06 on dist-test-slave-x4sn"
server_key: "cd0844174f3716baa0befd478f8744c2"
server_key_iv: "98132cf5b49384b32f658e630ac63a6d"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:06.783869 28373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:06.802023 28373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:06.809020 28373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:06.809306 28373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:06.810087 28373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:41:06.811218 28373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:41:06.811323 28373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:06.811416 28373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:41:06.811475 28373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:06.822686 28373 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:33429
I20260926 14:41:06.827136 28373 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:41:06.828356 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 28373
I20260926 14:41:06.828477 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:41:06.828811 16747 external_mini_cluster.cc:1428] Setting key e7226e3d651d3c908a94d76da5ad6ee8
I20260926 14:41:06.830495 28502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:33429 every 8 connection(s)
Sep 26 14:41:06 dist-test-slave-x4sn krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433666, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:06.848687 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.834152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:52549 (local address 127.16.90.254:32921)
0926 14:41:06.834265 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:06.834269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:06.834716 (+ 447us) server_negotiation.cc:407] Connection header received
0926 14:41:06.836040 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:06.836044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:06.836098 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:06.836153 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:06.837839 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.838568 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.840362 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.840702 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.840779 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.844279 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:06.844283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:06.844286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:06.844316 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:06.846059 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:06.846709 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:06.846713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:06.846715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:06.846809 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:06.847276 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:06.847279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:06.847281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:06.847531 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:06.847621 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:06.848391 (+ 770us) server_negotiation.cc:299] Negotiation successful
0926 14:41:06.848528 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:41:06.850648 28505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.833077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:32921 (local address 127.16.90.195:52549)
0926 14:41:06.834561 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:06.834622 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:06.835796 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:06.836588 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:06.836598 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:06.836970 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:06.837678 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.837697 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.838725 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.838729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.840222 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.840234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.840991 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.840995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.841195 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.842014 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:06.842036 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:06.844082 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:06.846192 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:06.846198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:06.846216 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:06.846586 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:06.846923 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:06.846926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:06.846928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:06.847123 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:06.847744 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:06.847750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:06.848084 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 14:41:06.849139 (+ 1055us) client_negotiation.cc:240] Negotiation successful
0926 14:41:06.849501 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1257,"thread_start_us":166,"threads_started":1}
I20260926 14:41:06.852349 28503 heartbeater.cc:347] Connected to a master server at 127.16.90.254:32921
I20260926 14:41:06.852594 28503 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:06.853142 28503 heartbeater.cc:511] Master 127.16.90.254:32921 requested a full tablet report, sending...
I20260926 14:41:06.854557 28016 ts_manager.cc:194] Registered new tserver with Master: 0c89fe97418149f29f65f9a80fad094f (127.16.90.195:33429)
I20260926 14:41:06.855352 28016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:52549
I20260926 14:41:06.856912 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:41:06.871649 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.861894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37618 (local address 127.16.90.254:32921)
0926 14:41:06.862008 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:06.862013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:06.862711 (+ 698us) server_negotiation.cc:407] Connection header received
0926 14:41:06.862890 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:06.862893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:06.862939 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:06.862987 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:06.864165 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.864887 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.866000 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.866291 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.866367 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.868205 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:06.868209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:06.868212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:06.868236 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:06.869809 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:06.870260 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:06.870264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:06.870265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:06.870345 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:06.870658 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:06.870661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:06.870663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:06.870871 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:06.870974 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:06.871343 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 14:41:06.871469 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:41:06.875576 28017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:41:06.880013 28017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:41:06.905565 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.891761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:37369 (local address 127.0.0.1:48470)
0926 14:41:06.897073 (+ 5312us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:06.897126 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:06.897245 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:06.900519 (+ 3274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:06.900522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:06.900540 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:41:06.900634 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.900640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.902217 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.902221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.903654 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.903687 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.904930 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.904934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.905058 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.905099 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 14:41:06.905264 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 14:41:06.905399 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5131,"thread_start_us":95,"threads_started":1}
I20260926 14:41:06.906659 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.891904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.16.90.193:37369)
0926 14:41:06.899868 (+ 7964us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:06.899875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:06.899899 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:41:06.899970 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:06.899975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:06.900226 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:06.900370 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:41:06.900800 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.902066 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.903816 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.904644 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.904750 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.906353 (+ 1603us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:06.906412 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 14:41:06.906513 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7853,"thread_start_us":85,"threads_started":1}
I20260926 14:41:06.908638 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.894844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33429 (local address 127.0.0.1:38740)
0926 14:41:06.897601 (+ 2757us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:06.897611 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:06.897751 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:06.900178 (+ 2427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:06.900181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:06.900198 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:41:06.900290 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.900297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.904029 (+ 3732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.904032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.905216 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.905228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.908247 (+ 3019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.908251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.908345 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.908368 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:41:06.908436 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:41:06.908505 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2709,"thread_start_us":81,"threads_started":1}
I20260926 14:41:06.909085 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.895220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38740 (local address 127.16.90.195:33429)
0926 14:41:06.896869 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:06.896880 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:06.897836 (+ 956us) server_negotiation.cc:407] Connection header received
0926 14:41:06.897918 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:06.897924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:06.898149 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:06.900013 (+ 1864us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:41:06.900811 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.903897 (+ 3086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.907085 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.907931 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.908038 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.908187 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:06.908828 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 14:41:06.908934 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1552,"thread_start_us":134,"threads_started":1}
I20260926 14:41:06.909840 28163 tablet_service.cc:1467] Processing CreateTablet for tablet 8758cb6f520d4a51965bf6a5be40d9ab (DEFAULT_TABLE table=test-table [id=775a146cd25741b58ea1e2b6d1271ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:41:06.911008 28163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8758cb6f520d4a51965bf6a5be40d9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:41:06.913240 28436 tablet_service.cc:1467] Processing CreateTablet for tablet 8758cb6f520d4a51965bf6a5be40d9ab (DEFAULT_TABLE table=test-table [id=775a146cd25741b58ea1e2b6d1271ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:41:06.914363 28436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8758cb6f520d4a51965bf6a5be40d9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:41:06.923691 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.893255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:41293 (local address 127.0.0.1:45462)
0926 14:41:06.897412 (+ 4157us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:06.897422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:06.897483 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:06.899463 (+ 1980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:06.899470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:06.899495 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:41:06.899739 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.899747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.905477 (+ 5730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.905480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.907897 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.907908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.909586 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.909590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.909677 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.909700 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:41:06.909758 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 14:41:06.909823 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4097,"thread_start_us":85,"threads_started":1}
I20260926 14:41:06.924499 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.893545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.16.90.194:41293)
0926 14:41:06.895867 (+ 2322us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:06.895878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:06.898283 (+ 2405us) server_negotiation.cc:407] Connection header received
0926 14:41:06.898367 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:06.898374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:06.898581 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:06.900458 (+ 1877us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:41:06.900607 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.905370 (+ 4763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.908068 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.909278 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.909381 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.909525 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:06.924206 (+ 14681us) server_negotiation.cc:299] Negotiation successful
0926 14:41:06.924352 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2210,"thread_start_us":133,"threads_started":1}
I20260926 14:41:06.928036 28522 tablet_bootstrap.cc:492] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c: Bootstrap starting.
I20260926 14:41:06.928651 28302 tablet_service.cc:1467] Processing CreateTablet for tablet 8758cb6f520d4a51965bf6a5be40d9ab (DEFAULT_TABLE table=test-table [id=775a146cd25741b58ea1e2b6d1271ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:41:06.929790 28302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8758cb6f520d4a51965bf6a5be40d9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:41:06.930490 28522 tablet_bootstrap.cc:654] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c: Neither blocks nor log segments found. Creating new log.
I20260926 14:41:06.931342 28522 log.cc:824] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:41:06.932128 28523 tablet_bootstrap.cc:492] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f: Bootstrap starting.
I20260926 14:41:06.934962 28523 tablet_bootstrap.cc:654] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f: Neither blocks nor log segments found. Creating new log.
I20260926 14:41:06.935793 28523 log.cc:824] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:41:06.938434 28526 tablet_bootstrap.cc:492] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f: Bootstrap starting.
I20260926 14:41:06.940855 28526 tablet_bootstrap.cc:654] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f: Neither blocks nor log segments found. Creating new log.
I20260926 14:41:06.942106 28526 log.cc:824] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:41:06.944245 28526 tablet_bootstrap.cc:492] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f: No bootstrap required, opened a new log
I20260926 14:41:06.944440 28526 ts_tablet_manager.cc:1397] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:41:06.945008 28522 tablet_bootstrap.cc:492] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c: No bootstrap required, opened a new log
I20260926 14:41:06.945209 28522 ts_tablet_manager.cc:1397] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20260926 14:41:06.948869 28526 raft_consensus.cc:348] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1872b4ac0b024d57801fad6520ab3f2c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37369 } } peers { permanent_uuid: "31efa6b771ee4931a9c528205b26850f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41293 } } peers { permanent_uuid: "0c89fe97418149f29f65f9a80fad094f" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33429 } }
I20260926 14:41:06.949147 28526 raft_consensus.cc:374] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:41:06.949208 28526 raft_consensus.cc:729] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31efa6b771ee4931a9c528205b26850f, State: Initialized, Role: FOLLOWER
I20260926 14:41:06.949293 28522 raft_consensus.cc:348] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1872b4ac0b024d57801fad6520ab3f2c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37369 } } peers { permanent_uuid: "31efa6b771ee4931a9c528205b26850f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41293 } } peers { permanent_uuid: "0c89fe97418149f29f65f9a80fad094f" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33429 } }
I20260926 14:41:06.949528 28522 raft_consensus.cc:374] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:41:06.949576 28522 raft_consensus.cc:729] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1872b4ac0b024d57801fad6520ab3f2c, State: Initialized, Role: FOLLOWER
I20260926 14:41:06.949712 28526 consensus_queue.cc:260] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1872b4ac0b024d57801fad6520ab3f2c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37369 } } peers { permanent_uuid: "31efa6b771ee4931a9c528205b26850f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41293 } } peers { permanent_uuid: "0c89fe97418149f29f65f9a80fad094f" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33429 } }
I20260926 14:41:06.950740 28526 ts_tablet_manager.cc:1428] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:41:06.951087 28368 heartbeater.cc:503] Master 127.16.90.254:32921 was elected leader, sending a full tablet report...
I20260926 14:41:06.951476 28522 consensus_queue.cc:260] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1872b4ac0b024d57801fad6520ab3f2c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37369 } } peers { permanent_uuid: "31efa6b771ee4931a9c528205b26850f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41293 } } peers { permanent_uuid: "0c89fe97418149f29f65f9a80fad094f" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33429 } }
I20260926 14:41:06.952548 28522 ts_tablet_manager.cc:1428] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 14:41:06.953717 28230 heartbeater.cc:503] Master 127.16.90.254:32921 was elected leader, sending a full tablet report...
I20260926 14:41:06.957391 28523 tablet_bootstrap.cc:492] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f: No bootstrap required, opened a new log
I20260926 14:41:06.957600 28523 ts_tablet_manager.cc:1397] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20260926 14:41:06.961643 28523 raft_consensus.cc:348] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1872b4ac0b024d57801fad6520ab3f2c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37369 } } peers { permanent_uuid: "31efa6b771ee4931a9c528205b26850f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41293 } } peers { permanent_uuid: "0c89fe97418149f29f65f9a80fad094f" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33429 } }
I20260926 14:41:06.961954 28523 raft_consensus.cc:374] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:41:06.962021 28523 raft_consensus.cc:729] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c89fe97418149f29f65f9a80fad094f, State: Initialized, Role: FOLLOWER
I20260926 14:41:06.963078 28523 consensus_queue.cc:260] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1872b4ac0b024d57801fad6520ab3f2c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37369 } } peers { permanent_uuid: "31efa6b771ee4931a9c528205b26850f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41293 } } peers { permanent_uuid: "0c89fe97418149f29f65f9a80fad094f" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33429 } }
I20260926 14:41:06.965633 28523 ts_tablet_manager.cc:1428] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 14:41:06.965977 28503 heartbeater.cc:503] Master 127.16.90.254:32921 was elected leader, sending a full tablet report...
I20260926 14:41:06.969524 28529 raft_consensus.cc:482] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:41:06.969693 28529 raft_consensus.cc:504] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1872b4ac0b024d57801fad6520ab3f2c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37369 } } peers { permanent_uuid: "31efa6b771ee4931a9c528205b26850f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41293 } } peers { permanent_uuid: "0c89fe97418149f29f65f9a80fad094f" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33429 } }
I20260926 14:41:06.970685 28529 leader_election.cc:288] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31efa6b771ee4931a9c528205b26850f (127.16.90.194:41293), 0c89fe97418149f29f65f9a80fad094f (127.16.90.195:33429)
I20260926 14:41:06.976033 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.971129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:37033 (local address 127.16.90.195:33429)
0926 14:41:06.971261 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:06.971265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:06.971971 (+ 706us) server_negotiation.cc:407] Connection header received
0926 14:41:06.972126 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:06.972129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:06.972168 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:06.972211 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:41:06.972547 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.973237 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.974707 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.975299 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.975384 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.975433 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:06.975834 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 14:41:06.975908 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:41:06.976033 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.971350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:45225 (local address 127.16.90.194:41293)
0926 14:41:06.971464 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:06.971468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:06.971701 (+ 233us) server_negotiation.cc:407] Connection header received
0926 14:41:06.971812 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:06.971815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:06.971865 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:06.971928 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:41:06.972319 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.973083 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.974707 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.975300 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.975384 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.975433 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:06.975835 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 14:41:06.975908 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:41:06.976445 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.970875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:41293 (local address 127.16.90.193:45225)
0926 14:41:06.971594 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:06.971613 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:06.971729 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:06.972037 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:06.972039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:06.972053 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:41:06.972204 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.972211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.973221 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.973223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.974568 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.974578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.975539 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.975542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.975641 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.975715 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 14:41:06.976224 (+ 509us) client_negotiation.cc:240] Negotiation successful
0926 14:41:06.976300 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":43,"thread_start_us":194,"threads_started":1}
I20260926 14:41:06.976783 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:06.970926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33429 (local address 127.16.90.193:37033)
0926 14:41:06.971906 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:06.971917 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:06.972024 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:06.972325 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:06.972327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:06.972341 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:41:06.972438 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.972443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.973370 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.973373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.974568 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:06.974578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:06.975566 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:06.975569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:06.975716 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:06.975732 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 14:41:06.976433 (+ 701us) client_negotiation.cc:240] Negotiation successful
0926 14:41:06.976643 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":44,"thread_start_us":419,"threads_started":1}
I20260926 14:41:06.977026 28322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8758cb6f520d4a51965bf6a5be40d9ab" candidate_uuid: "1872b4ac0b024d57801fad6520ab3f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31efa6b771ee4931a9c528205b26850f" is_pre_election: true
I20260926 14:41:06.977350 28322 raft_consensus.cc:2454] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1872b4ac0b024d57801fad6520ab3f2c in term 0.
I20260926 14:41:06.977969 28117 leader_election.cc:302] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1872b4ac0b024d57801fad6520ab3f2c, 31efa6b771ee4931a9c528205b26850f; no voters:
I20260926 14:41:06.978369 28441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8758cb6f520d4a51965bf6a5be40d9ab" candidate_uuid: "1872b4ac0b024d57801fad6520ab3f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c89fe97418149f29f65f9a80fad094f" is_pre_election: true
I20260926 14:41:06.978466 28529 raft_consensus.cc:2781] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:41:06.978545 28529 raft_consensus.cc:482] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:41:06.978586 28529 raft_consensus.cc:3037] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:41:06.978689 28441 raft_consensus.cc:2454] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1872b4ac0b024d57801fad6520ab3f2c in term 0.
I20260926 14:41:06.979720 28529 raft_consensus.cc:504] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1872b4ac0b024d57801fad6520ab3f2c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37369 } } peers { permanent_uuid: "31efa6b771ee4931a9c528205b26850f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41293 } } peers { permanent_uuid: "0c89fe97418149f29f65f9a80fad094f" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33429 } }
I20260926 14:41:06.980046 28529 leader_election.cc:288] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [CANDIDATE]: Term 1 election: Requested vote from peers 31efa6b771ee4931a9c528205b26850f (127.16.90.194:41293), 0c89fe97418149f29f65f9a80fad094f (127.16.90.195:33429)
I20260926 14:41:06.980432 28322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8758cb6f520d4a51965bf6a5be40d9ab" candidate_uuid: "1872b4ac0b024d57801fad6520ab3f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31efa6b771ee4931a9c528205b26850f"
I20260926 14:41:06.980473 28441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8758cb6f520d4a51965bf6a5be40d9ab" candidate_uuid: "1872b4ac0b024d57801fad6520ab3f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c89fe97418149f29f65f9a80fad094f"
I20260926 14:41:06.980535 28322 raft_consensus.cc:3037] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:41:06.980608 28441 raft_consensus.cc:3037] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:41:06.981469 28322 raft_consensus.cc:2454] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1872b4ac0b024d57801fad6520ab3f2c in term 1.
I20260926 14:41:06.981734 28117 leader_election.cc:302] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1872b4ac0b024d57801fad6520ab3f2c, 31efa6b771ee4931a9c528205b26850f; no voters:
I20260926 14:41:06.981889 28529 raft_consensus.cc:2781] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:41:06.981966 28441 raft_consensus.cc:2454] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1872b4ac0b024d57801fad6520ab3f2c in term 1.
I20260926 14:41:06.982101 28529 raft_consensus.cc:686] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [term 1 LEADER]: Becoming Leader. State: Replica: 1872b4ac0b024d57801fad6520ab3f2c, State: Running, Role: LEADER
I20260926 14:41:06.982425 28529 consensus_queue.cc:237] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1872b4ac0b024d57801fad6520ab3f2c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37369 } } peers { permanent_uuid: "31efa6b771ee4931a9c528205b26850f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41293 } } peers { permanent_uuid: "0c89fe97418149f29f65f9a80fad094f" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33429 } }
I20260926 14:41:06.985229 28017 catalog_manager.cc:5495] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1872b4ac0b024d57801fad6520ab3f2c (127.16.90.193). New cstate: current_term: 1 leader_uuid: "1872b4ac0b024d57801fad6520ab3f2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1872b4ac0b024d57801fad6520ab3f2c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31efa6b771ee4931a9c528205b26850f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c89fe97418149f29f65f9a80fad094f" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33429 } health_report { overall_health: UNKNOWN } } }
I20260926 14:41:07.006361 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:07.002202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.16.90.193:37369)
0926 14:41:07.002322 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:07.002326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:07.002407 (+ 81us) server_negotiation.cc:407] Connection header received
0926 14:41:07.002699 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:07.002703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:07.002780 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:07.002897 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:41:07.003286 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:07.003872 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:07.004719 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:07.004983 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:07.005071 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:07.005393 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:41:07.005868 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:41:07.006006 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:07.006146 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 14:41:07.006203 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:41:07.012099 28441 raft_consensus.cc:1261] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f [term 1 FOLLOWER]: Refusing update from remote peer 1872b4ac0b024d57801fad6520ab3f2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:41:07.012099 28322 raft_consensus.cc:1261] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f [term 1 FOLLOWER]: Refusing update from remote peer 1872b4ac0b024d57801fad6520ab3f2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:41:07.012823 28529 consensus_queue.cc:1035] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31efa6b771ee4931a9c528205b26850f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 41293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:41:07.013269 28533 consensus_queue.cc:1035] T 8758cb6f520d4a51965bf6a5be40d9ab P 1872b4ac0b024d57801fad6520ab3f2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c89fe97418149f29f65f9a80fad094f" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:41:07.026625 28537 mvcc.cc:205] Tried to move back new op lower bound from 7333616300074569728 to 7333616299962400768. Current Snapshot: MvccSnapshot[applied={T|T < 7333616300074569728}]
I20260926 14:41:07.040016 28016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:41:07.040185 28016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:41:07.042548 28016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c7d546817264800b1d8b618b0150b38: Sending DeleteTablet for 3 replicas of tablet 8758cb6f520d4a51965bf6a5be40d9ab
I20260926 14:41:07.043428 28436 tablet_service.cc:1514] Processing DeleteTablet for tablet 8758cb6f520d4a51965bf6a5be40d9ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:41:07 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:38740
I20260926 14:41:07.043454 28302 tablet_service.cc:1514] Processing DeleteTablet for tablet 8758cb6f520d4a51965bf6a5be40d9ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:41:07 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:45462
I20260926 14:41:07.043985 28545 tablet_replica.cc:330] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f: stopping tablet replica
I20260926 14:41:07.044188 28163 tablet_service.cc:1514] Processing DeleteTablet for tablet 8758cb6f520d4a51965bf6a5be40d9ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:41:07 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:48470
I20260926 14:41:07.044447 28545 raft_consensus.cc:2229] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:41:07.044822 28545 raft_consensus.cc:2258] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:41:07.045006 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 28098
I20260926 14:41:07.046785 28546 tablet_replica.cc:330] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f: stopping tablet replica
I20260926 14:41:07.047099 28546 raft_consensus.cc:2229] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:41:07.047431 28546 raft_consensus.cc:2258] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:41:07.049190 28546 ts_tablet_manager.cc:1905] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:41:07.049324 28545 ts_tablet_manager.cc:1905] T 8758cb6f520d4a51965bf6a5be40d9ab P 31efa6b771ee4931a9c528205b26850f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 14:41:07.053194 28000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:37369 (error 108)
I20260926 14:41:07.054190 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 28238
I20260926 14:41:07.055557 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:07.055166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:37369 (local address 127.0.0.1:48488)
0926 14:41:07.055342 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:07.055415 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:37369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260926 14:41:07.055781 28000 catalog_manager.cc:4584] TS 1872b4ac0b024d57801fad6520ab3f2c (127.16.90.193:37369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8758cb6f520d4a51965bf6a5be40d9ab: Network error: Client connection negotiation failed: client connection to 127.16.90.193:37369: connect: Connection refused (error 111)
I20260926 14:41:07.059167 28546 ts_tablet_manager.cc:1918] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:41:07.059276 28546 log.cc:1196] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/8758cb6f520d4a51965bf6a5be40d9ab
I20260926 14:41:07.059602 28546 ts_tablet_manager.cc:1939] T 8758cb6f520d4a51965bf6a5be40d9ab P 0c89fe97418149f29f65f9a80fad094f: Deleting consensus metadata
I20260926 14:41:07.060984 28000 catalog_manager.cc:4841] TS 0c89fe97418149f29f65f9a80fad094f (127.16.90.195:33429): tablet 8758cb6f520d4a51965bf6a5be40d9ab (table test-table [id=775a146cd25741b58ea1e2b6d1271ef7]) successfully deleted
I20260926 14:41:07.063016 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 28373
I20260926 14:41:07.063289 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:07.063009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:41293 (local address 127.0.0.1:45464)
0926 14:41:07.063151 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:07.063208 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.194:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 14:41:07.071045 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 27984
I20260926 14:41:07.076876 16747 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 14:41:07.411609 16747 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 14:41:07.411767 16747 mini_ranger.cc:66] Stopping Ranger...
I20260926 14:41:07.746628 16747 mini_ranger.cc:68] Stopped Ranger
2026-09-26 14:41:07.746 UTC [27569] LOG: received smart shutdown request
2026-09-26 14:41:07.750 UTC [27569] LOG: background worker "logical replication launcher" (PID 27578) exited with exit code 1
2026-09-26 14:41:07.750 UTC [27573] LOG: shutting down
2026-09-26 14:41:07.898 UTC [27569] LOG: database system is shut down
2026-09-26T14:41:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44084 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:41:08 dist-test-slave-x4sn krb5kdc[28557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:41:08 dist-test-slave-x4sn krb5kdc[28557](info): set up 2 sockets
Sep 26 14:41:08 dist-test-slave-x4sn krb5kdc[28557](info): commencing operation
krb5kdc: starting...
W20260926 14:41:08.949074 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:41:09 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:41:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:41:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.90.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.90.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/rangeradmin_127.16.90.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/rangeradmin_127.16.90.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.90.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.90.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/rangerlookup_127.16.90.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/rangerlookup_127.16.90.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.90.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.90.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/HTTP_127.16.90.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/HTTP_127.16.90.212@KRBTEST.COM.keytab.
I20260926 14:41:09.110641 16747 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasks9vtzz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/postgres -l logfile start
2026-09-26 14:41:11.443 UTC [28587] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 14:41:11.443 UTC [28587] LOG: listening on IPv4 address "127.16.90.212", port 48629
2026-09-26 14:41:11.446 UTC [28587] LOG: listening on Unix socket "/tmp/.s.PGSQL.48629"
2026-09-26 14:41:11.473 UTC [28590] LOG: database system was shut down at 2026-09-26 14:41:10 UTC
2026-09-26 14:41:11.477 UTC [28587] LOG: database system is ready to accept connections
I20260926 14:41:12.345063 16747 mini_postgres.cc:96] Postgres bound to 48629
2026-09-26 14:41:12.351 UTC [28598] FATAL: database "slave" does not exist
127.16.90.212:48629 - accepting connections
I20260926 14:41:12.351953 16747 mini_ranger.cc:161] Starting Ranger...
I20260926 14:41:12.366268 16747 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 14:41:12.801663 16747 mini_ranger.cc:87] Created ranger Postgres database
I20260926 14:41:12.801759 16747 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-admin
2026-09-26 14:41:13,060 [I] DB FLAVOR :POSTGRES
2026-09-26 14:41:13,061 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:41:13,061 [I] Checking connection..
2026-09-26 14:41:13,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 14:41:13,310 [I] Checking connection passed.
2026-09-26 14:41:13,310 [I] --------- Verifying version history table ---------
2026-09-26 14:41:13,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:41:13,562 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:41:13,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:41:13,806 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:41:13,807 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 14:41:13,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 14:41:14.036 UTC [28662] WARNING: there is no transaction in progress
2026-09-26 14:41:14,042 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 14:41:14,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:41:14,277 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 14:41:14,278 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:41:14,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 14:41:14,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 14:41:14,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-x4sn.c.gcp-upstream.internal','N') ;"
2026-09-26 14:41:14,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:41:15,224 [I] Table x_portal_user does not exist in database ranger
2026-09-26 14:41:15,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:41:15,474 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 14:41:15,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:41:15,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:41:15,963 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 14:41:15,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 14:41:16.847 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:16.863 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:16.877 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:16.891 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:16.905 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:16.924 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:16.973 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:16.978 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:16.990 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:17.000 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:17.009 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:17.021 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:17.030 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:17.036 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:17.041 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:17.047 UTC [28771] WARNING: there is no transaction in progress
2026-09-26 14:41:17,614 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 14:41:17,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 14:41:17,862 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 14:41:17,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:41:18,089 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 14:41:18,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:41:18,324 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 14:41:18,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:41:18,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:41:18,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-x4sn.c.gcp-upstream.internal';"
2026-09-26 14:41:19,014 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 14:41:19,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 14:41:19,257 [I] DB_PATCHES have already been applied
I20260926 14:41:19.263149 16747 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-admin:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasks9vtzz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 14:41:19.263216 16747 mini_ranger.cc:193] Using host: 127.16.90.212
I20260926 14:41:19.266499 16747 mini_ranger.cc:227] Ranger admin URL: http://127.16.90.212:47669
Sep 26, 2026 2:41:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:41:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:41:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:41:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:41:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:41:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:41:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 14:41:20.269788 16747 mini_ranger.cc:160] Time spent starting Ranger: real 7.918s user 0.002s sys 0.007s
Sep 26, 2026 2:41:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:41:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 2:41:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:41:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:41:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:41:40 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 14:41:41.369566 16747 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.90.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.90.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/rangerkms_127.16.90.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.90.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/rangerkms_127.16.90.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.90.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.90.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.90.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/HTTP_127.16.90.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.90.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/HTTP_127.16.90.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 14:41:41 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433701, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 14:41:41.447834 16747 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 14:41:41.456564 16747 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 14:41:42.171741 16747 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 14:41:42.171813 16747 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-kms
I20260926 14:41:42.171829 16747 mini_ranger_kms.cc:227] Using postgres at 127.16.90.212:48629
2026-09-26 14:41:42,324 [I] DB FLAVOR :POSTGRES
2026-09-26 14:41:42,324 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:41:42,324 [I] Checking connection
2026-09-26 14:41:42,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 14:41:42,546 [I] connection success
2026-09-26 14:41:42,546 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 14:41:42,546 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 14:41:42,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 14:41:42,778 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 14:41:42,778 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:41:42,778 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 14:41:42,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.90.212:48629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 14:41:43,018 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 14:41:43.469950 16747 mini_ranger_kms.cc:310] Created kms service
I20260926 14:41:43.656492 16747 mini_ranger_kms.cc:326] Created kudu user
I20260926 14:41:43.697196 16747 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 14:41:43 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433701, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:44.065717 16747 mini_ranger_kms.cc:384] Added ranger policy
I20260926 14:41:44.065871 16747 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-kms:/tmp/dist-test-tasks9vtzz/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasks9vtzz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 14:41:44.065891 16747 mini_ranger_kms.cc:240] Using host: 127.16.90.212
I20260926 14:41:44.070071 16747 mini_ranger_kms.cc:276] Ranger KMS PID: 28951
I20260926 14:41:44.070150 16747 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.90.212:58513
Sep 26, 2026 2:41:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:41:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:41:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:41:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:41:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:41:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 14:41:46.081020 16747 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.633s user 0.015s sys 0.122s
I20260926 14:41:46.081166 16747 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 14:41:46.081256 16747 mini_ranger_kms.cc:401] 127.16.90.212:58513/kms/v1/keys
Sep 26 14:41:46 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790433706, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.90.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 14:41:46 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433706, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.90.212@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
Sep 26 14:41:47 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433706, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.90.212@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
Sep 26 14:41:49 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:41:49.392355 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:38341
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:38341
--ranger_config_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.90.212:58513/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:41:49.501330 29002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:49.501608 29002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:49.501647 29002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:49.505766 29002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:41:49.505841 29002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:49.505859 29002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:49.505874 29002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:41:49.505887 29002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:41:49.511971 29002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.90.212:58513/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:38341
--ranger_config_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:38341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:49.513085 29002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:49.514129 29002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:49.522971 29008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:49.523113 29002 server_base.cc:1034] running on GCE node
W20260926 14:41:49.523010 29007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:49.523281 29010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:49.523933 29002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:49.525035 29002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:49.526208 29002 hybrid_clock.cc:648] HybridClock initialized: now 1790433709526194 us; error 32 us; skew 500 ppm
Sep 26 14:41:49 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433709, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:49.532133 29002 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:41:49.533119 29002 webserver.cc:460] Webserver started at http://127.16.90.254:39145/ using document root <none> and password file <none>
I20260926 14:41:49.533635 29002 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:49.533690 29002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:41:49.533880 29002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:41:49 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433709, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:49.588127 29002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "07be369f6a2f4dbead1e2292fd405617"
format_stamp: "Formatted at 2026-09-26 14:41:49 on dist-test-slave-x4sn"
server_key: "cd4c536ec695e350a6f07756dd5b4da7"
server_key_iv: "698be85f9cfed2ad5aa0850d8cec270d"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:49.588846 29002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "07be369f6a2f4dbead1e2292fd405617"
format_stamp: "Formatted at 2026-09-26 14:41:49 on dist-test-slave-x4sn"
server_key: "cd4c536ec695e350a6f07756dd5b4da7"
server_key_iv: "698be85f9cfed2ad5aa0850d8cec270d"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:49.592662 29002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:41:49.610026 29017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:49.611440 29002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:41:49.611569 29002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "07be369f6a2f4dbead1e2292fd405617"
format_stamp: "Formatted at 2026-09-26 14:41:49 on dist-test-slave-x4sn"
server_key: "cd4c536ec695e350a6f07756dd5b4da7"
server_key_iv: "698be85f9cfed2ad5aa0850d8cec270d"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:49.611739 29002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:49.624274 29002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:49.627060 29002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:49.627233 29002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:49.634964 29002 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:38341
I20260926 14:41:49.634982 29069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:38341 every 8 connection(s)
I20260926 14:41:49.636000 29002 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:41:49.638725 29070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:41:49.639245 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 29002
I20260926 14:41:49.639367 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:41:49.639736 16747 external_mini_cluster.cc:1428] Setting key e7667944ecbfc97a8cda5d7cf771678d
I20260926 14:41:49.645219 29070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617: Bootstrap starting.
I20260926 14:41:49.647768 29070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617: Neither blocks nor log segments found. Creating new log.
I20260926 14:41:49.648700 29070 log.cc:824] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:41:49 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:49.650830 29070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617: No bootstrap required, opened a new log
I20260926 14:41:49.655112 29070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07be369f6a2f4dbead1e2292fd405617" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38341 } }
I20260926 14:41:49.655398 29070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:41:49.655481 29070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07be369f6a2f4dbead1e2292fd405617, State: Initialized, Role: FOLLOWER
I20260926 14:41:49.656024 29073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:49.641171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.16.90.254:38341)
0926 14:41:49.641712 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:49.641732 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:49.641777 (+ 45us) server_negotiation.cc:407] Connection header received
0926 14:41:49.642645 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:49.642671 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:49.643114 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:49.643473 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:49.644419 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:49.645900 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:49.647094 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:49.647845 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:49.648038 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:49.650920 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:49.650927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:49.650944 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:49.650975 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:49.652868 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:49.653346 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:49.653352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:49.653358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:49.653489 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:49.653755 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:49.653758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:49.653760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:49.654275 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:49.654476 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:49.654745 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 14:41:49.655018 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":187,"threads_started":1}
I20260926 14:41:49.656009 29070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07be369f6a2f4dbead1e2292fd405617" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38341 } }
I20260926 14:41:49.656169 29070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:41:49.656225 29070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:41:49.656342 29070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:41:49.657476 29070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07be369f6a2f4dbead1e2292fd405617" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38341 } }
I20260926 14:41:49.657819 29070 leader_election.cc:302] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07be369f6a2f4dbead1e2292fd405617; no voters:
I20260926 14:41:49.658130 29070 leader_election.cc:288] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:41:49.658224 29075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:41:49.658458 29075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 1 LEADER]: Becoming Leader. State: Replica: 07be369f6a2f4dbead1e2292fd405617, State: Running, Role: LEADER
I20260926 14:41:49.658793 29075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07be369f6a2f4dbead1e2292fd405617" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38341 } }
I20260926 14:41:49.659494 29070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:41:49.659976 29075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07be369f6a2f4dbead1e2292fd405617. Latest consensus state: current_term: 1 leader_uuid: "07be369f6a2f4dbead1e2292fd405617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07be369f6a2f4dbead1e2292fd405617" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38341 } } }
I20260926 14:41:49.660158 29075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [sys.catalog]: This master's current role is: LEADER
I20260926 14:41:49.660403 29075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07be369f6a2f4dbead1e2292fd405617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07be369f6a2f4dbead1e2292fd405617" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38341 } } }
I20260926 14:41:49.660544 29075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [sys.catalog]: This master's current role is: LEADER
I20260926 14:41:49.660837 29081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:41:49.664476 29081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:41:49.665000 29070 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 14:41:49.670694 29081 catalog_manager.cc:1347] Generated new cluster ID: 65511fc6b26843348a4d6d7b5b32ed35
I20260926 14:41:49.670849 29081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:41:49.686621 29081 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:41:49.688038 29081 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:41:49.700851 29081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617: Generated new TSK 0
I20260926 14:41:49.701871 29081 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:41:50.470813 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.90.212:58513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38341
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:41:50.631018 29115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:50.631459 29115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:50.631573 29115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:50.637465 29115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:50.637607 29115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:50.637741 29115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:41:50.645340 29115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.90.212:58513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:50.646817 29115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:50.647965 29115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:50.657202 29123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:50.657202 29120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:50.657737 29121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:50.658187 29115 server_base.cc:1034] running on GCE node
I20260926 14:41:50.658566 29115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:50.659233 29115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:50.660390 29115 hybrid_clock.cc:648] HybridClock initialized: now 1790433710660365 us; error 36 us; skew 500 ppm
Sep 26 14:41:50 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433710, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:50.663161 29115 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:41:50.664124 29115 webserver.cc:460] Webserver started at http://127.16.90.193:43793/ using document root <none> and password file <none>
I20260926 14:41:50.664834 29115 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:50.664975 29115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:41:50.665284 29115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:41:50 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433710, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:50.683133 29115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "a9c67381f0b742029365346f5b6e0d55"
format_stamp: "Formatted at 2026-09-26 14:41:50 on dist-test-slave-x4sn"
server_key: "450477be5ae70567f125cd138df03d89"
server_key_iv: "2db15ba97e6a7ceefd30a81f58b3cc0b"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:50.683933 29115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "a9c67381f0b742029365346f5b6e0d55"
format_stamp: "Formatted at 2026-09-26 14:41:50 on dist-test-slave-x4sn"
server_key: "450477be5ae70567f125cd138df03d89"
server_key_iv: "2db15ba97e6a7ceefd30a81f58b3cc0b"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:50.696437 29115 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.001s
I20260926 14:41:50.713253 29132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:50.715040 29115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:41:50.715241 29115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "a9c67381f0b742029365346f5b6e0d55"
format_stamp: "Formatted at 2026-09-26 14:41:50 on dist-test-slave-x4sn"
server_key: "450477be5ae70567f125cd138df03d89"
server_key_iv: "2db15ba97e6a7ceefd30a81f58b3cc0b"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:50.715480 29115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:50.731523 29115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:50.735075 29115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:50.735260 29115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:50.736003 29115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:41:50.737020 29115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:41:50.737082 29115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:50.737133 29115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:41:50.737166 29115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:50.747730 29246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:35437 every 8 connection(s)
I20260926 14:41:50.747783 29115 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:35437
I20260926 14:41:50.748852 29115 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:41:50.754868 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 29115
I20260926 14:41:50.755015 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:41:50.755309 16747 external_mini_cluster.cc:1428] Setting key 6f2e5d9470cd2f4ddb0fe739a7da17a3
Sep 26 14:41:50 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433710, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:50.764926 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:50.750625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:55775 (local address 127.16.90.254:38341)
0926 14:41:50.750876 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:50.750880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:50.751700 (+ 820us) server_negotiation.cc:407] Connection header received
0926 14:41:50.752690 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:50.752693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:50.752752 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:50.752840 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:50.754124 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:50.755020 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:50.756913 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:50.757275 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:50.757354 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:50.760420 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:50.760424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:50.760426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:50.760449 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:50.762316 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:50.762938 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:50.762941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:50.762944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:50.763037 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:50.763436 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:50.763439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:50.763441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:50.763824 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:50.763921 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:50.764616 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 14:41:50.764747 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
I20260926 14:41:50.766964 29250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:50.751081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38341 (local address 127.16.90.193:55775)
0926 14:41:50.751527 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:50.751579 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:50.752478 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:50.753018 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:50.753025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:50.753301 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:50.753936 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:50.753964 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:50.755287 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:50.755291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:50.756776 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:50.756787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:50.757565 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:50.757568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:50.757759 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:50.758389 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:50.758409 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:50.760235 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:50.762454 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:50.762460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:50.762475 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:50.762821 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:50.763145 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:50.763148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:50.763150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:50.763331 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:50.763991 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:50.763998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:50.764316 (+ 318us) client_negotiation.cc:769] Sending connection context
0926 14:41:50.765574 (+ 1258us) client_negotiation.cc:240] Negotiation successful
0926 14:41:50.765914 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
I20260926 14:41:50.768603 29247 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38341
I20260926 14:41:50.768839 29247 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:50.769371 29247 heartbeater.cc:511] Master 127.16.90.254:38341 requested a full tablet report, sending...
I20260926 14:41:50.771065 29034 ts_manager.cc:194] Registered new tserver with Master: a9c67381f0b742029365346f5b6e0d55 (127.16.90.193:35437)
I20260926 14:41:50.772681 29034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:55775
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:41:50.804615 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.90.212:58513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38341
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:41:50 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790433710, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 14:41:50.940944 29255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:50.941358 29255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:50.941430 29255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:50.947378 29255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:50.947453 29255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:50.947536 29255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:41:50.955291 29255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.90.212:58513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:50.956727 29255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:50.958093 29255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:50.969412 29264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:50.969514 29261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:50.969415 29262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:50.969862 29255 server_base.cc:1034] running on GCE node
I20260926 14:41:50.970386 29255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:50.971050 29255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:50.972251 29255 hybrid_clock.cc:648] HybridClock initialized: now 1790433710972224 us; error 37 us; skew 500 ppm
Sep 26 14:41:50 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433710, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:50.975131 29255 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:41:50.976111 29255 webserver.cc:460] Webserver started at http://127.16.90.194:45761/ using document root <none> and password file <none>
I20260926 14:41:50.976660 29255 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:50.976717 29255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:41:50.976915 29255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:41:50 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433710, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:50.994900 29255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "67fbc21792654e3ba739badbcafb9800"
format_stamp: "Formatted at 2026-09-26 14:41:50 on dist-test-slave-x4sn"
server_key: "94e0b4d52c378910e55cff405f4b7203"
server_key_iv: "8e3ad48779383ddfdf226c7e5b3ab893"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:50.995877 29255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "67fbc21792654e3ba739badbcafb9800"
format_stamp: "Formatted at 2026-09-26 14:41:50 on dist-test-slave-x4sn"
server_key: "94e0b4d52c378910e55cff405f4b7203"
server_key_iv: "8e3ad48779383ddfdf226c7e5b3ab893"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:51.005228 29255 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.000s
I20260926 14:41:51.025717 29271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:51.027952 29255 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260926 14:41:51.028117 29255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "67fbc21792654e3ba739badbcafb9800"
format_stamp: "Formatted at 2026-09-26 14:41:50 on dist-test-slave-x4sn"
server_key: "94e0b4d52c378910e55cff405f4b7203"
server_key_iv: "8e3ad48779383ddfdf226c7e5b3ab893"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:51.028287 29255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:51.050863 29255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:51.063279 29255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:51.063576 29255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:51.064433 29255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:41:51.065578 29255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:41:51.065683 29255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:51.065779 29255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:41:51.065841 29255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:51.079701 29255 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:33517
I20260926 14:41:51.080979 29255 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:41:51.082352 29385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:33517 every 8 connection(s)
I20260926 14:41:51.082916 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 29255
I20260926 14:41:51.083063 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:41:51.083360 16747 external_mini_cluster.cc:1428] Setting key beca9eff061da33acf76d56a75615829
Sep 26 14:41:51 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433710, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:51.103404 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:51.086170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:52995 (local address 127.16.90.254:38341)
0926 14:41:51.086289 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:51.086294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:51.086764 (+ 470us) server_negotiation.cc:407] Connection header received
0926 14:41:51.088043 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:51.088047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:51.088096 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:51.088157 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:51.089801 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:51.090631 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:51.092377 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:51.092729 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:51.092809 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:51.098765 (+ 5956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:51.098770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:51.098774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:51.098805 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:51.100658 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:51.101396 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:51.101400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:51.101402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:51.101496 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:51.101962 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:51.101967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:51.101968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:51.102234 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:51.102322 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:51.103075 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 14:41:51.103231 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:41:51.105568 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:51.085074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38341 (local address 127.16.90.194:52995)
0926 14:41:51.086607 (+ 1533us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:51.086665 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:51.087806 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:51.088560 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:51.088569 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:51.088923 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:51.089620 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:51.089642 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:51.090797 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:51.090800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:51.092230 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:51.092241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:51.093013 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:51.093016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:51.093234 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:51.096705 (+ 3471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:51.096729 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:51.098511 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:51.100831 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:51.100838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:51.100856 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:51.101263 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:51.101609 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:51.101612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:51.101614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:51.101840 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:51.102390 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:51.102396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:51.102762 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 14:41:51.104029 (+ 1267us) client_negotiation.cc:240] Negotiation successful
0926 14:41:51.104424 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"thread_start_us":161,"threads_started":1}
I20260926 14:41:51.109140 29386 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38341
I20260926 14:41:51.109416 29386 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:51.110034 29386 heartbeater.cc:511] Master 127.16.90.254:38341 requested a full tablet report, sending...
I20260926 14:41:51.111270 29034 ts_manager.cc:194] Registered new tserver with Master: 67fbc21792654e3ba739badbcafb9800 (127.16.90.194:33517)
I20260926 14:41:51.112078 29034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:52995
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:41:51.143529 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.90.212:58513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38341
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:41:51.311908 29391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:51.312755 29391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:51.312906 29391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:51.319275 29391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:51.319406 29391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:51.319535 29391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:41:51.327346 29391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.90.212:58513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:51.328881 29391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:51.330188 29391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:41:51.343793 29391 server_base.cc:1034] running on GCE node
W20260926 14:41:51.343936 29397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:51.348141 29399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:51.351836 29396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:51.352487 29391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:51.353307 29391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:51.362180 29391 hybrid_clock.cc:648] HybridClock initialized: now 1790433711362097 us; error 90 us; skew 500 ppm
Sep 26 14:41:51 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433711, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:51.365897 29391 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:41:51.367012 29391 webserver.cc:460] Webserver started at http://127.16.90.195:36335/ using document root <none> and password file <none>
I20260926 14:41:51.367712 29391 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:51.367782 29391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:41:51.367995 29391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:41:51 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433711, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:51.385607 29391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "849d008645cb4a77b6760e183893be58"
format_stamp: "Formatted at 2026-09-26 14:41:51 on dist-test-slave-x4sn"
server_key: "93bf5b35eb227e0fe16582abdaf46f68"
server_key_iv: "270c887a2a91529771bc91d4e30ede03"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:51.386437 29391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "849d008645cb4a77b6760e183893be58"
format_stamp: "Formatted at 2026-09-26 14:41:51 on dist-test-slave-x4sn"
server_key: "93bf5b35eb227e0fe16582abdaf46f68"
server_key_iv: "270c887a2a91529771bc91d4e30ede03"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:51.397164 29391 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.000s sys 0.004s
Sep 26 14:41:51 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433710, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:51.415973 29406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:51.417704 29391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:41:51.417830 29391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "849d008645cb4a77b6760e183893be58"
format_stamp: "Formatted at 2026-09-26 14:41:51 on dist-test-slave-x4sn"
server_key: "93bf5b35eb227e0fe16582abdaf46f68"
server_key_iv: "270c887a2a91529771bc91d4e30ede03"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:51.417958 29391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:51.447184 29391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:51.452854 29391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:51.453069 29391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:51.454625 29391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:41:51.456168 29391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:41:51.456259 29391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:51.456326 29391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:41:51.456362 29391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:51.467170 29391 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:34725
I20260926 14:41:51.468444 29391 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:41:51.477253 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 29391
I20260926 14:41:51.477384 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:41:51.477687 16747 external_mini_cluster.cc:1428] Setting key b995711fc1085425cb4fa881f0de4542
I20260926 14:41:51.496183 29519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:34725 every 8 connection(s)
Sep 26 14:41:51 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433711, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:51.534380 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:51.499817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:41139 (local address 127.16.90.254:38341)
0926 14:41:51.499936 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:51.499940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:51.507695 (+ 7755us) server_negotiation.cc:407] Connection header received
0926 14:41:51.511106 (+ 3411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:51.511110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:51.511168 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:51.511222 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:51.513055 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:51.513895 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:51.518873 (+ 4978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:51.519237 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:51.519314 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:51.528308 (+ 8994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:51.528313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:51.528316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:51.528349 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:51.530084 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:51.530797 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:51.530801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:51.530802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:51.530896 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:51.531332 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:51.531336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:51.531338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:51.531625 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:51.533342 (+ 1717us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:51.534092 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 14:41:51.534218 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:41:51.536724 29522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:51.498592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38341 (local address 127.16.90.195:41139)
0926 14:41:51.507443 (+ 8851us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:51.507541 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:51.510848 (+ 3307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:51.511734 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:51.511745 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:51.512132 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:51.512887 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:51.512909 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:51.514067 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:51.514070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:51.516703 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:51.516714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:51.519556 (+ 2842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:51.519560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:51.521505 (+ 1945us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:51.522361 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:51.522383 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:51.528068 (+ 5685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:51.530234 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:51.530240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:51.530258 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:51.530672 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:51.531008 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:51.531011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:51.531013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:51.531213 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:51.533411 (+ 2198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:51.533418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:51.533797 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 14:41:51.534842 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0926 14:41:51.535219 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8560,"thread_start_us":191,"threads_started":1}
I20260926 14:41:51.538214 29520 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38341
I20260926 14:41:51.538479 29520 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:51.539041 29520 heartbeater.cc:511] Master 127.16.90.254:38341 requested a full tablet report, sending...
I20260926 14:41:51.540081 29034 ts_manager.cc:194] Registered new tserver with Master: 849d008645cb4a77b6760e183893be58 (127.16.90.195:34725)
I20260926 14:41:51.540783 29034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:41139
I20260926 14:41:51.544988 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:41:51.545830 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 29115
I20260926 14:41:51.553966 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 29255
I20260926 14:41:51.562791 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 29391
I20260926 14:41:51.569890 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 29002
I20260926 14:41:51.576327 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:38341
--webserver_interface=127.16.90.254
--webserver_port=39145
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:38341
--ranger_config_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.90.212:58513/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.90.212:58513/kms with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:41:51.685650 29523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:51.685916 29523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:51.685971 29523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:51.689934 29523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:41:51.690059 29523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:51.690099 29523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:51.690122 29523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:41:51.690150 29523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:41:51.695554 29523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.90.212:58513/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:38341
--ranger_config_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:38341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=39145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:51.696600 29523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:51.697566 29523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:51.705691 29528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:51.705674 29529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:51.705725 29523 server_base.cc:1034] running on GCE node
W20260926 14:41:51.705873 29531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:51.706439 29523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:51.707545 29523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:51.708729 29523 hybrid_clock.cc:648] HybridClock initialized: now 1790433711708710 us; error 34 us; skew 500 ppm
Sep 26 14:41:51 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433711, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:51.711441 29523 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:41:51.712379 29523 webserver.cc:460] Webserver started at http://127.16.90.254:39145/ using document root <none> and password file <none>
I20260926 14:41:51.712899 29523 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:51.712983 29523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:41:51 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433711, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:51.773442 29523 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:41:51.775749 29539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:51.777141 29523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:41:51.777266 29523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "07be369f6a2f4dbead1e2292fd405617"
format_stamp: "Formatted at 2026-09-26 14:41:49 on dist-test-slave-x4sn"
server_key: "cd4c536ec695e350a6f07756dd5b4da7"
server_key_iv: "698be85f9cfed2ad5aa0850d8cec270d"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:51.777823 29523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:51.805075 29523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:51.807989 29523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:51.808164 29523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:51.815263 29523 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:38341
I20260926 14:41:51.815290 29592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:38341 every 8 connection(s)
I20260926 14:41:51.816331 29523 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:41:51.820601 29593 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:41:51.822173 29593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617: Bootstrap starting.
I20260926 14:41:51.823220 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 29523
I20260926 14:41:51.823720 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.90.212:58513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:35437
--local_ip_for_outbound_sockets=127.16.90.193
--tserver_master_addrs=127.16.90.254:38341
--webserver_port=43793
--webserver_interface=127.16.90.193
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.90.212:58513/kms with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:41:51.837862 29593 log.cc:824] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617: Log is configured to *not* fsync() on all Append() calls
I20260926 14:41:51.844522 29593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:41:51.844846 29593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617: Bootstrap complete.
I20260926 14:41:51.848598 29593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07be369f6a2f4dbead1e2292fd405617" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38341 } }
I20260926 14:41:51.848956 29593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07be369f6a2f4dbead1e2292fd405617, State: Initialized, Role: FOLLOWER
I20260926 14:41:51.849414 29593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07be369f6a2f4dbead1e2292fd405617" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38341 } }
I20260926 14:41:51.849543 29593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:41:51.849607 29593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:41:51.849699 29593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:41:51.851454 29593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07be369f6a2f4dbead1e2292fd405617" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38341 } }
I20260926 14:41:51.851799 29593 leader_election.cc:302] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07be369f6a2f4dbead1e2292fd405617; no voters:
I20260926 14:41:51.852077 29593 leader_election.cc:288] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:41:51.852169 29597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:41:51.852335 29597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [term 2 LEADER]: Becoming Leader. State: Replica: 07be369f6a2f4dbead1e2292fd405617, State: Running, Role: LEADER
I20260926 14:41:51.852676 29597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07be369f6a2f4dbead1e2292fd405617" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38341 } }
I20260926 14:41:51.853296 29593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:41:51.853540 29597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07be369f6a2f4dbead1e2292fd405617. Latest consensus state: current_term: 2 leader_uuid: "07be369f6a2f4dbead1e2292fd405617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07be369f6a2f4dbead1e2292fd405617" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38341 } } }
I20260926 14:41:51.853645 29597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [sys.catalog]: This master's current role is: LEADER
I20260926 14:41:51.854257 29598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "07be369f6a2f4dbead1e2292fd405617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07be369f6a2f4dbead1e2292fd405617" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38341 } } }
I20260926 14:41:51.854432 29598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617 [sys.catalog]: This master's current role is: LEADER
I20260926 14:41:51.855448 29601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:41:51.855739 29593 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 14:41:51.861023 29601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:41:51.862321 29601 catalog_manager.cc:1259] Loaded cluster ID: 65511fc6b26843348a4d6d7b5b32ed35
I20260926 14:41:51.862368 29601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:41:51.864917 29601 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:41:51.865646 29601 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617: Loaded TSK: 0
I20260926 14:41:51.866894 29601 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:41:51.963033 29595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:51.963516 29595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:51.963639 29595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:51.969550 29595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:51.969625 29595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:51.969709 29595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:41:51.977229 29595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.90.212:58513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:35437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=43793
--tserver_master_addrs=127.16.90.254:38341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:51.978636 29595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:51.979933 29595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:51.991573 29633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:51.991644 29631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:51.992390 29630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:51.992889 29595 server_base.cc:1034] running on GCE node
I20260926 14:41:51.993326 29595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:51.994109 29595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:51.995301 29595 hybrid_clock.cc:648] HybridClock initialized: now 1790433711995274 us; error 38 us; skew 500 ppm
Sep 26 14:41:51 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433711, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:51.998714 29595 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:41:51.999801 29595 webserver.cc:460] Webserver started at http://127.16.90.193:43793/ using document root <none> and password file <none>
I20260926 14:41:52.000447 29595 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:52.000560 29595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:41:52 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433711, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:52.041460 29595 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.003s sys 0.000s
I20260926 14:41:52.044576 29641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:52.045930 29595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:41:52.046054 29595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "a9c67381f0b742029365346f5b6e0d55"
format_stamp: "Formatted at 2026-09-26 14:41:50 on dist-test-slave-x4sn"
server_key: "450477be5ae70567f125cd138df03d89"
server_key_iv: "2db15ba97e6a7ceefd30a81f58b3cc0b"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:52.046624 29595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:52.070904 29595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:52.073796 29595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:52.073976 29595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:52.074580 29595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:41:52.075510 29595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:41:52.075614 29595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:52.075760 29595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:41:52.075834 29595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:52.086416 29595 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:35437
I20260926 14:41:52.088165 29595 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:41:52.090165 29754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:35437 every 8 connection(s)
I20260926 14:41:52.098400 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 29595
I20260926 14:41:52.098883 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.90.212:58513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:33517
--local_ip_for_outbound_sockets=127.16.90.194
--tserver_master_addrs=127.16.90.254:38341
--webserver_port=45761
--webserver_interface=127.16.90.194
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.90.212:58513/kms with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:41:52 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433711, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:52.112654 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:52.092749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38341 (local address 127.16.90.193:35545)
0926 14:41:52.093204 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:52.093245 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:52.094055 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:52.095635 (+ 1580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:52.095644 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:52.095989 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:52.096629 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:52.096649 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.098056 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.098059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:52.099312 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:52.099323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.100313 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.100318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:52.100532 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:52.103366 (+ 2834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:52.103386 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:52.106021 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:52.108943 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:52.108949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:52.108965 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:52.109303 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:52.109732 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:52.109736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:52.109738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:52.109908 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:52.110699 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:52.110705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:52.111009 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 14:41:52.111213 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 14:41:52.111551 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":139,"threads_started":1}
I20260926 14:41:52.114662 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:52.092332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:35545 (local address 127.16.90.254:38341)
0926 14:41:52.092935 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:52.092952 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:52.093361 (+ 409us) server_negotiation.cc:407] Connection header received
0926 14:41:52.094468 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:52.094486 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:52.094783 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:52.095117 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:52.096801 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.097906 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.099447 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.100145 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.100293 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:52.106264 (+ 5971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:52.106271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:52.106288 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:52.106321 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:52.108800 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:52.109445 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:52.109452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:52.109457 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:52.109620 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:52.110022 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:52.110025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:52.110027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:52.110521 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:52.110645 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:52.113442 (+ 2797us) server_negotiation.cc:299] Negotiation successful
0926 14:41:52.113743 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":117,"threads_started":1}
I20260926 14:41:52.130599 29755 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38341
I20260926 14:41:52.130897 29755 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:52.131469 29755 heartbeater.cc:511] Master 127.16.90.254:38341 requested a full tablet report, sending...
W20260926 14:41:52.133904 29755 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:38341 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 14:41:52.134047 29755 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:52.134222 29755 heartbeater.cc:511] Master 127.16.90.254:38341 requested a full tablet report, sending...
I20260926 14:41:52.135125 29755 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:52.135303 29755 heartbeater.cc:511] Master 127.16.90.254:38341 requested a full tablet report, sending...
W20260926 14:41:52.136108 29755 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:41:52.229660 29759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:52.230008 29759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:52.230072 29759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:52.234490 29759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:52.234549 29759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:52.234625 29759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:41:52.240195 29759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.90.212:58513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:33517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=45761
--tserver_master_addrs=127.16.90.254:38341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:52.241468 29759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:52.242758 29759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:52.257932 29767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:52.258178 29759 server_base.cc:1034] running on GCE node
W20260926 14:41:52.258497 29764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:52.259063 29765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:52.259584 29759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:52.260491 29759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:52.261722 29759 hybrid_clock.cc:648] HybridClock initialized: now 1790433712261642 us; error 86 us; skew 500 ppm
Sep 26 14:41:52 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433712, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:52.265362 29759 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:41:52.266566 29759 webserver.cc:460] Webserver started at http://127.16.90.194:45761/ using document root <none> and password file <none>
I20260926 14:41:52.267261 29759 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:52.267356 29759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:41:52 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433712, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:52.291272 29759 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:41:52.293752 29776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:52.295059 29759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:41:52.295255 29759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "67fbc21792654e3ba739badbcafb9800"
format_stamp: "Formatted at 2026-09-26 14:41:50 on dist-test-slave-x4sn"
server_key: "94e0b4d52c378910e55cff405f4b7203"
server_key_iv: "8e3ad48779383ddfdf226c7e5b3ab893"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:52.296001 29759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:52.323868 29759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:52.327075 29759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:52.327266 29759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:52.327934 29759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:41:52.328764 29759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:41:52.328812 29759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:52.328863 29759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:41:52.328891 29759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:52.338603 29759 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:33517
I20260926 14:41:52.338702 29890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:33517 every 8 connection(s)
I20260926 14:41:52.339591 29759 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:41:52.348465 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 29759
I20260926 14:41:52.348963 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.90.212:58513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:34725
--local_ip_for_outbound_sockets=127.16.90.195
--tserver_master_addrs=127.16.90.254:38341
--webserver_port=36335
--webserver_interface=127.16.90.195
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.90.212:58513/kms with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:41:52 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433712, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:52.355079 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:52.341729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38341 (local address 127.16.90.194:56005)
0926 14:41:52.342164 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:52.342214 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:52.343162 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:52.343919 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:52.343925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:52.344186 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:52.344712 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:52.344727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.345551 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.345556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:52.346945 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:52.346956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.347480 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.347485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:52.347707 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:52.348341 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:52.348363 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:52.349971 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:52.351894 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:52.351900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:52.351915 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:52.352265 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:52.352582 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:52.352585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:52.352587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:52.352758 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:52.353240 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:52.353247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:52.353544 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 14:41:52.353742 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 14:41:52.354054 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":106,"threads_started":1}
I20260926 14:41:52.356079 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:52.341356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:56005 (local address 127.16.90.254:38341)
0926 14:41:52.341525 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:52.341530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:52.342309 (+ 779us) server_negotiation.cc:407] Connection header received
0926 14:41:52.343407 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:52.343411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:52.343462 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:52.343515 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:52.344845 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.345380 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.347092 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.347327 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.347407 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:52.350168 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:52.350173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:52.350176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:52.350206 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:52.351764 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:52.352386 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:52.352389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:52.352391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:52.352483 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:52.352867 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:52.352870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:52.352872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:52.353086 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:52.353185 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:52.355803 (+ 2618us) server_negotiation.cc:299] Negotiation successful
0926 14:41:52.355932 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:41:52.356971 29891 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38341
I20260926 14:41:52.357226 29891 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:52.357739 29891 heartbeater.cc:511] Master 127.16.90.254:38341 requested a full tablet report, sending...
W20260926 14:41:52.359333 29891 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:38341 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 14:41:52.359447 29891 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:52.359614 29891 heartbeater.cc:511] Master 127.16.90.254:38341 requested a full tablet report, sending...
I20260926 14:41:52.360381 29891 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:52.360538 29891 heartbeater.cc:511] Master 127.16.90.254:38341 requested a full tablet report, sending...
W20260926 14:41:52.361192 29891 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:41:52.513087 29894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:52.513459 29894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:52.513528 29894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:52.519450 29894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:52.519531 29894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:52.519618 29894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:41:52.527266 29894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.90.212:58513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:34725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=36335
--tserver_master_addrs=127.16.90.254:38341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:52.528672 29894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:52.530036 29894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:52.541496 29899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:52.541915 29894 server_base.cc:1034] running on GCE node
W20260926 14:41:52.542608 29900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:52.543004 29902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:52.543598 29894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:52.544492 29894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:52.545706 29894 hybrid_clock.cc:648] HybridClock initialized: now 1790433712545676 us; error 49 us; skew 500 ppm
Sep 26 14:41:52 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433712, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:52.549856 29894 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:41:52.550825 29894 webserver.cc:460] Webserver started at http://127.16.90.195:36335/ using document root <none> and password file <none>
I20260926 14:41:52.551365 29894 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:52.551445 29894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:41:52 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433712, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:52.573686 29894 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:41:52.578089 29910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:52.579533 29894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:41:52.579751 29894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "849d008645cb4a77b6760e183893be58"
format_stamp: "Formatted at 2026-09-26 14:41:51 on dist-test-slave-x4sn"
server_key: "93bf5b35eb227e0fe16582abdaf46f68"
server_key_iv: "270c887a2a91529771bc91d4e30ede03"
server_key_version: "kuduclusterkey@0"
I20260926 14:41:52.580330 29894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:52.612118 29894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:52.616159 29894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:52.616370 29894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:52.617057 29894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:41:52.618083 29894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:41:52.618141 29894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:52.618197 29894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:41:52.618230 29894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:52.628850 29894 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:34725
I20260926 14:41:52.628903 30031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:34725 every 8 connection(s)
I20260926 14:41:52.630151 29894 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:41:52.632757 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 29894
Sep 26 14:41:52 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433712, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:52.642400 30035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:52.633345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.16.90.254:38341)
0926 14:41:52.634012 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:52.634023 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:52.634038 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:41:52.634079 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:52.634081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:52.634133 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:52.634229 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:52.635140 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.635944 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.637188 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.637533 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.637631 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:52.638966 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:52.638970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:52.638973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:52.638997 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:52.640467 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:52.640926 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:52.640928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:52.640929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:52.641002 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:52.641268 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:52.641271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:52.641273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:52.641486 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:52.641677 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:52.642118 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 14:41:52.642233 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":103,"threads_started":1}
I20260926 14:41:52.645223 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:52.632191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:40003 (local address 127.16.90.254:38341)
0926 14:41:52.632316 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:52.632319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:52.633782 (+ 1463us) server_negotiation.cc:407] Connection header received
0926 14:41:52.634762 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:52.634765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:52.634811 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:52.634868 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:52.636089 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.636713 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.638052 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.638357 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.638448 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:52.641383 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:52.641386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:52.641388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:52.641407 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:52.642902 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:52.643426 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:52.643430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:52.643432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:52.643532 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:52.643948 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:52.643988 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:52.643990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:52.644211 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:52.644332 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:52.644930 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 14:41:52.645060 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:41:52.646222 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:52.632734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38341 (local address 127.16.90.195:40003)
0926 14:41:52.633585 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:52.633643 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:52.634571 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:52.635053 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:52.635059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:52.635360 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:52.635984 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:52.636002 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.636826 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.636830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:52.637908 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:52.637917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:52.638585 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:52.638588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:52.638773 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:52.639372 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:52.639391 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:52.641238 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:52.643026 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:52.643031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:52.643045 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:52.643319 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:52.643649 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:52.643651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:52.643653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:52.643853 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:52.644324 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:52.644331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:52.644615 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 14:41:52.644833 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 14:41:52.645202 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":164,"threads_started":1}
I20260926 14:41:52.648520 30032 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38341
I20260926 14:41:52.648773 30032 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:52.649322 30032 heartbeater.cc:511] Master 127.16.90.254:38341 requested a full tablet report, sending...
I20260926 14:41:52.650835 29557 ts_manager.cc:194] Registered new tserver with Master: 849d008645cb4a77b6760e183893be58 (127.16.90.195:34725)
I20260926 14:41:52.652508 29557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:40003
Sep 26 14:41:52 dist-test-slave-x4sn krb5kdc[28557](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790433712, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:53.136350 29755 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:53.136652 29755 heartbeater.cc:511] Master 127.16.90.254:38341 requested a full tablet report, sending...
I20260926 14:41:53.137537 29557 ts_manager.cc:194] Registered new tserver with Master: a9c67381f0b742029365346f5b6e0d55 (127.16.90.193:35437)
I20260926 14:41:53.138339 29557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:35545
Sep 26 14:41:53 dist-test-slave-x4sn krb5kdc[28557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433712, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for HTTP/127.16.90.212@KRBTEST.COM
I20260926 14:41:53.361388 29891 heartbeater.cc:464] Registering TS with master...
I20260926 14:41:53.361680 29891 heartbeater.cc:511] Master 127.16.90.254:38341 requested a full tablet report, sending...
I20260926 14:41:53.362466 29557 ts_manager.cc:194] Registered new tserver with Master: 67fbc21792654e3ba739badbcafb9800 (127.16.90.194:33517)
I20260926 14:41:53.363209 29557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:56005
I20260926 14:41:53.363324 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:41:53.374073 30047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.365107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.16.90.254:38341)
0926 14:41:53.365319 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:53.365323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:53.365730 (+ 407us) server_negotiation.cc:407] Connection header received
0926 14:41:53.365837 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:53.365840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:53.365885 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:53.365983 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:53.366743 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.367550 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.368636 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.368981 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.369087 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:53.370591 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:53.370595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:53.370598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:53.370623 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:53.372377 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:53.372833 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:53.372837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:53.372839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:53.372926 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:53.373179 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:53.373182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:53.373184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:53.373384 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:53.373506 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:53.373716 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 14:41:53.373898 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
I20260926 14:41:53.377960 29557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:41:53.382675 29557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:41:53.404414 30054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.393911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:33517 (local address 127.0.0.1:41576)
0926 14:41:53.395374 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:53.395424 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:53.395550 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:53.396350 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:53.396356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:53.396379 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:41:53.396526 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:53.396532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.400865 (+ 4333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.400868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:53.402162 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:53.402171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.403920 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.403923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:53.404033 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:53.404074 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 14:41:53.404150 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 14:41:53.404259 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260926 14:41:53.405414 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.395107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.16.90.194:33517)
0926 14:41:53.395841 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:53.395848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:53.395871 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:41:53.395943 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:53.395948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:53.396180 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:53.398982 (+ 2802us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:41:53.399160 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.400750 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.402963 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.403800 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.404992 (+ 1192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:53.405138 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:53.405189 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 14:41:53.405279 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":88,"threads_started":1}
I20260926 14:41:53.408665 29825 tablet_service.cc:1467] Processing CreateTablet for tablet 972b77e357484c73afc5273843bd1216 (DEFAULT_TABLE table=test-table [id=b5c29acb108c4dd59558b2c21d4b4212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:41:53.409864 29825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972b77e357484c73afc5273843bd1216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:41:53.410391 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.394677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:34725 (local address 127.0.0.1:50084)
0926 14:41:53.399816 (+ 5139us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:53.399828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:53.399901 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:53.401594 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:53.401597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:53.401612 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:41:53.401698 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:53.401719 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.407703 (+ 5984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.407706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:53.408766 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:53.408777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.410038 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.410042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:53.410125 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:53.410147 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:41:53.410205 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 14:41:53.410266 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5083,"thread_start_us":90,"threads_started":1}
I20260926 14:41:53.412043 30057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.395782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.16.90.195:34725)
0926 14:41:53.401042 (+ 5260us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:53.401049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:53.401071 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:41:53.401138 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:53.401143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:53.401329 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:53.401466 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:41:53.405772 (+ 4306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.407539 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.408905 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.409731 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.409829 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:53.411775 (+ 1946us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:53.411826 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 14:41:53.411914 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5159,"thread_start_us":71,"threads_started":1}
I20260926 14:41:53.414917 29965 tablet_service.cc:1467] Processing CreateTablet for tablet 972b77e357484c73afc5273843bd1216 (DEFAULT_TABLE table=test-table [id=b5c29acb108c4dd59558b2c21d4b4212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:41:53.415970 29965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972b77e357484c73afc5273843bd1216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:41:53.419158 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.394104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:35437 (local address 127.0.0.1:49822)
0926 14:41:53.399548 (+ 5444us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:53.399562 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:53.399638 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:53.400843 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:53.400846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:53.400863 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:41:53.401861 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:53.401866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.403110 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.403113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:53.404240 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:53.404248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.405397 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.405400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:53.405487 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:53.405512 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:41:53.405577 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:41:53.405638 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5383,"mutex_wait_us":919,"thread_start_us":107,"threads_started":1}
I20260926 14:41:53.419791 30056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.395368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.16.90.193:35437)
0926 14:41:53.400152 (+ 4784us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:53.400159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:53.400193 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:41:53.400263 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:53.400268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:53.400562 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:53.400692 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:41:53.401969 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.402997 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.404359 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.405110 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.405218 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:53.405337 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:53.419470 (+ 14133us) server_negotiation.cc:299] Negotiation successful
0926 14:41:53.419637 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4680,"thread_start_us":127,"threads_started":1}
I20260926 14:41:53.422574 29673 tablet_service.cc:1467] Processing CreateTablet for tablet 972b77e357484c73afc5273843bd1216 (DEFAULT_TABLE table=test-table [id=b5c29acb108c4dd59558b2c21d4b4212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:41:53.423611 29673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972b77e357484c73afc5273843bd1216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:41:53.424317 30058 tablet_bootstrap.cc:492] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800: Bootstrap starting.
I20260926 14:41:53.426697 30058 tablet_bootstrap.cc:654] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800: Neither blocks nor log segments found. Creating new log.
I20260926 14:41:53.427536 30058 log.cc:824] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800: Log is configured to *not* fsync() on all Append() calls
I20260926 14:41:53.430153 30058 tablet_bootstrap.cc:492] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800: No bootstrap required, opened a new log
I20260926 14:41:53.430358 30058 ts_tablet_manager.cc:1397] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:41:53.432914 30059 tablet_bootstrap.cc:492] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58: Bootstrap starting.
I20260926 14:41:53.434193 30058 raft_consensus.cc:348] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fbc21792654e3ba739badbcafb9800" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33517 } } peers { permanent_uuid: "849d008645cb4a77b6760e183893be58" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34725 } } peers { permanent_uuid: "a9c67381f0b742029365346f5b6e0d55" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 35437 } }
I20260926 14:41:53.434463 30058 raft_consensus.cc:374] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:41:53.434511 30058 raft_consensus.cc:729] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67fbc21792654e3ba739badbcafb9800, State: Initialized, Role: FOLLOWER
I20260926 14:41:53.435096 30058 consensus_queue.cc:260] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fbc21792654e3ba739badbcafb9800" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33517 } } peers { permanent_uuid: "849d008645cb4a77b6760e183893be58" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34725 } } peers { permanent_uuid: "a9c67381f0b742029365346f5b6e0d55" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 35437 } }
I20260926 14:41:53.436124 30058 ts_tablet_manager.cc:1428] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 14:41:53.436342 30059 tablet_bootstrap.cc:654] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58: Neither blocks nor log segments found. Creating new log.
I20260926 14:41:53.436468 29891 heartbeater.cc:503] Master 127.16.90.254:38341 was elected leader, sending a full tablet report...
I20260926 14:41:53.437906 30059 log.cc:824] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58: Log is configured to *not* fsync() on all Append() calls
I20260926 14:41:53.438817 30061 tablet_bootstrap.cc:492] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55: Bootstrap starting.
I20260926 14:41:53.442096 30061 tablet_bootstrap.cc:654] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55: Neither blocks nor log segments found. Creating new log.
I20260926 14:41:53.443804 30061 log.cc:824] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55: Log is configured to *not* fsync() on all Append() calls
I20260926 14:41:53.446388 30061 tablet_bootstrap.cc:492] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55: No bootstrap required, opened a new log
I20260926 14:41:53.447791 30061 ts_tablet_manager.cc:1397] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260926 14:41:53.447978 30059 tablet_bootstrap.cc:492] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58: No bootstrap required, opened a new log
I20260926 14:41:53.448187 30059 ts_tablet_manager.cc:1397] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260926 14:41:53.455684 30061 raft_consensus.cc:348] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fbc21792654e3ba739badbcafb9800" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33517 } } peers { permanent_uuid: "849d008645cb4a77b6760e183893be58" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34725 } } peers { permanent_uuid: "a9c67381f0b742029365346f5b6e0d55" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 35437 } }
I20260926 14:41:53.455643 30059 raft_consensus.cc:348] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fbc21792654e3ba739badbcafb9800" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33517 } } peers { permanent_uuid: "849d008645cb4a77b6760e183893be58" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34725 } } peers { permanent_uuid: "a9c67381f0b742029365346f5b6e0d55" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 35437 } }
I20260926 14:41:53.455976 30059 raft_consensus.cc:374] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:41:53.456058 30059 raft_consensus.cc:729] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849d008645cb4a77b6760e183893be58, State: Initialized, Role: FOLLOWER
I20260926 14:41:53.456085 30061 raft_consensus.cc:374] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:41:53.456212 30061 raft_consensus.cc:729] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c67381f0b742029365346f5b6e0d55, State: Initialized, Role: FOLLOWER
I20260926 14:41:53.456640 30059 consensus_queue.cc:260] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fbc21792654e3ba739badbcafb9800" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33517 } } peers { permanent_uuid: "849d008645cb4a77b6760e183893be58" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34725 } } peers { permanent_uuid: "a9c67381f0b742029365346f5b6e0d55" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 35437 } }
I20260926 14:41:53.456683 30061 consensus_queue.cc:260] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fbc21792654e3ba739badbcafb9800" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33517 } } peers { permanent_uuid: "849d008645cb4a77b6760e183893be58" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34725 } } peers { permanent_uuid: "a9c67381f0b742029365346f5b6e0d55" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 35437 } }
I20260926 14:41:53.457928 30059 ts_tablet_manager.cc:1428] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58: Time spent starting tablet: real 0.010s user 0.001s sys 0.008s
I20260926 14:41:53.457929 30061 ts_tablet_manager.cc:1428] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260926 14:41:53.458290 30032 heartbeater.cc:503] Master 127.16.90.254:38341 was elected leader, sending a full tablet report...
I20260926 14:41:53.458315 29755 heartbeater.cc:503] Master 127.16.90.254:38341 was elected leader, sending a full tablet report...
I20260926 14:41:53.467378 30062 raft_consensus.cc:482] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:41:53.467608 30062 raft_consensus.cc:504] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fbc21792654e3ba739badbcafb9800" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33517 } } peers { permanent_uuid: "849d008645cb4a77b6760e183893be58" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34725 } } peers { permanent_uuid: "a9c67381f0b742029365346f5b6e0d55" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 35437 } }
I20260926 14:41:53.469017 30062 leader_election.cc:288] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 849d008645cb4a77b6760e183893be58 (127.16.90.195:34725), a9c67381f0b742029365346f5b6e0d55 (127.16.90.193:35437)
I20260926 14:41:53.474990 30056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.469713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:39357 (local address 127.16.90.193:35437)
0926 14:41:53.469845 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:53.469850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:53.470585 (+ 735us) server_negotiation.cc:407] Connection header received
0926 14:41:53.470785 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:53.470788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:53.470835 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:53.470881 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:41:53.471564 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.472374 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.473822 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.474369 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.474455 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:53.474503 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:53.474802 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 14:41:53.474873 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:41:53.475485 30057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.469575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:41245 (local address 127.16.90.195:34725)
0926 14:41:53.469699 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:53.469704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:53.470707 (+ 1003us) server_negotiation.cc:407] Connection header received
0926 14:41:53.470762 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:53.470765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:53.470809 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:53.470880 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:41:53.471987 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.472736 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.474227 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.474812 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.474907 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:53.474954 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:53.475294 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 14:41:53.475372 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:41:53.475703 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.469313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:35437 (local address 127.16.90.194:39357)
0926 14:41:53.470470 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:53.470483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:53.470653 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:53.471019 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:53.471022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:53.471039 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:41:53.471433 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:53.471443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.472498 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.472502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:53.473690 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:53.473701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.474589 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.474593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:53.474684 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:53.474704 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:41:53.475335 (+ 631us) client_negotiation.cc:240] Negotiation successful
0926 14:41:53.475406 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"thread_start_us":154,"threads_started":1}
I20260926 14:41:53.475904 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.469225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:34725 (local address 127.16.90.194:41245)
0926 14:41:53.470054 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:53.470068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:53.470199 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:53.471573 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:53.471576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:53.471596 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:41:53.471852 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:53.471862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.472908 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.472912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:53.474085 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:53.474096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.475052 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.475055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:53.475153 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:53.475174 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:41:53.475696 (+ 522us) client_negotiation.cc:240] Negotiation successful
0926 14:41:53.475769 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":104,"threads_started":1}
I20260926 14:41:53.476373 29693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972b77e357484c73afc5273843bd1216" candidate_uuid: "67fbc21792654e3ba739badbcafb9800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9c67381f0b742029365346f5b6e0d55" is_pre_election: true
I20260926 14:41:53.476459 29985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972b77e357484c73afc5273843bd1216" candidate_uuid: "67fbc21792654e3ba739badbcafb9800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849d008645cb4a77b6760e183893be58" is_pre_election: true
I20260926 14:41:53.476665 29693 raft_consensus.cc:2454] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67fbc21792654e3ba739badbcafb9800 in term 0.
I20260926 14:41:53.476732 29985 raft_consensus.cc:2454] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67fbc21792654e3ba739badbcafb9800 in term 0.
I20260926 14:41:53.477202 29778 leader_election.cc:302] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67fbc21792654e3ba739badbcafb9800, 849d008645cb4a77b6760e183893be58; no voters:
I20260926 14:41:53.477690 30062 raft_consensus.cc:2781] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:41:53.477787 30062 raft_consensus.cc:482] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:41:53.477838 30062 raft_consensus.cc:3037] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:41:53.479187 30062 raft_consensus.cc:504] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fbc21792654e3ba739badbcafb9800" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33517 } } peers { permanent_uuid: "849d008645cb4a77b6760e183893be58" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34725 } } peers { permanent_uuid: "a9c67381f0b742029365346f5b6e0d55" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 35437 } }
I20260926 14:41:53.479565 30062 leader_election.cc:288] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [CANDIDATE]: Term 1 election: Requested vote from peers 849d008645cb4a77b6760e183893be58 (127.16.90.195:34725), a9c67381f0b742029365346f5b6e0d55 (127.16.90.193:35437)
I20260926 14:41:53.479983 29985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972b77e357484c73afc5273843bd1216" candidate_uuid: "67fbc21792654e3ba739badbcafb9800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849d008645cb4a77b6760e183893be58"
I20260926 14:41:53.480109 29985 raft_consensus.cc:3037] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:41:53.480233 29693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972b77e357484c73afc5273843bd1216" candidate_uuid: "67fbc21792654e3ba739badbcafb9800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9c67381f0b742029365346f5b6e0d55"
I20260926 14:41:53.480346 29693 raft_consensus.cc:3037] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:41:53.481422 29985 raft_consensus.cc:2454] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67fbc21792654e3ba739badbcafb9800 in term 1.
I20260926 14:41:53.481611 29693 raft_consensus.cc:2454] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67fbc21792654e3ba739badbcafb9800 in term 1.
I20260926 14:41:53.481905 29778 leader_election.cc:302] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67fbc21792654e3ba739badbcafb9800, 849d008645cb4a77b6760e183893be58; no voters:
I20260926 14:41:53.482107 30062 raft_consensus.cc:2781] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:41:53.482342 30062 raft_consensus.cc:686] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 1 LEADER]: Becoming Leader. State: Replica: 67fbc21792654e3ba739badbcafb9800, State: Running, Role: LEADER
I20260926 14:41:53.482715 30062 consensus_queue.cc:237] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fbc21792654e3ba739badbcafb9800" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33517 } } peers { permanent_uuid: "849d008645cb4a77b6760e183893be58" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34725 } } peers { permanent_uuid: "a9c67381f0b742029365346f5b6e0d55" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 35437 } }
I20260926 14:41:53.486380 29557 catalog_manager.cc:5495] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67fbc21792654e3ba739badbcafb9800 (127.16.90.194). New cstate: current_term: 1 leader_uuid: "67fbc21792654e3ba739badbcafb9800" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fbc21792654e3ba739badbcafb9800" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849d008645cb4a77b6760e183893be58" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9c67381f0b742029365346f5b6e0d55" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 35437 } health_report { overall_health: UNKNOWN } } }
I20260926 14:41:53.500481 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.495833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.16.90.194:33517)
0926 14:41:53.495980 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:53.495985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:53.496000 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:41:53.496084 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:53.496087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:53.496172 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:53.496294 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:41:53.496576 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.497325 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.498491 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:53.498814 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:53.498898 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:53.499318 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:41:53.499917 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:41:53.500044 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:53.500224 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 14:41:53.500292 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:41:53.509824 29693 raft_consensus.cc:1261] T 972b77e357484c73afc5273843bd1216 P a9c67381f0b742029365346f5b6e0d55 [term 1 FOLLOWER]: Refusing update from remote peer 67fbc21792654e3ba739badbcafb9800: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:41:53.510059 29985 raft_consensus.cc:1261] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58 [term 1 FOLLOWER]: Refusing update from remote peer 67fbc21792654e3ba739badbcafb9800: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:41:53.510890 30069 consensus_queue.cc:1035] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c67381f0b742029365346f5b6e0d55" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:41:53.511058 30062 consensus_queue.cc:1035] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849d008645cb4a77b6760e183893be58" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:41:53.522236 30072 mvcc.cc:205] Tried to move back new op lower bound from 7333616490526543872 to 7333616490428162048. Current Snapshot: MvccSnapshot[applied={T|T < 7333616490526543872}]
I20260926 14:41:53.522571 30073 mvcc.cc:205] Tried to move back new op lower bound from 7333616490526543872 to 7333616490428162048. Current Snapshot: MvccSnapshot[applied={T|T < 7333616490526543872}]
I20260926 14:41:53.529542 30074 mvcc.cc:205] Tried to move back new op lower bound from 7333616490526543872 to 7333616490428162048. Current Snapshot: MvccSnapshot[applied={T|T < 7333616490526543872}]
I20260926 14:41:53.537352 29557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:41:53.537563 29557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:41:53.540802 29557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07be369f6a2f4dbead1e2292fd405617: Sending DeleteTablet for 3 replicas of tablet 972b77e357484c73afc5273843bd1216
I20260926 14:41:53.541790 29673 tablet_service.cc:1514] Processing DeleteTablet for tablet 972b77e357484c73afc5273843bd1216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:41:53 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:49822
I20260926 14:41:53.541934 29825 tablet_service.cc:1514] Processing DeleteTablet for tablet 972b77e357484c73afc5273843bd1216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:41:53 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:41576
I20260926 14:41:53.542387 29965 tablet_service.cc:1514] Processing DeleteTablet for tablet 972b77e357484c73afc5273843bd1216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:41:53 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:50084
I20260926 14:41:53.543355 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 29595
I20260926 14:41:53.552106 30083 tablet_replica.cc:330] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58: stopping tablet replica
I20260926 14:41:53.552471 30083 raft_consensus.cc:2229] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:41:53.552831 30083 raft_consensus.cc:2258] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:41:53.551872 30082 tablet_replica.cc:330] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800: stopping tablet replica
I20260926 14:41:53.553330 30082 raft_consensus.cc:2229] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:41:53.553723 30082 raft_consensus.cc:2258] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:41:53.555238 30082 ts_tablet_manager.cc:1905] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:41:53.557966 30082 ts_tablet_manager.cc:1918] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:41:53.558065 30082 log.cc:1196] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/972b77e357484c73afc5273843bd1216
I20260926 14:41:53.558390 30082 ts_tablet_manager.cc:1939] T 972b77e357484c73afc5273843bd1216 P 67fbc21792654e3ba739badbcafb9800: Deleting consensus metadata
I20260926 14:41:53.559695 29541 catalog_manager.cc:4841] TS 67fbc21792654e3ba739badbcafb9800 (127.16.90.194:33517): tablet 972b77e357484c73afc5273843bd1216 (table test-table [id=b5c29acb108c4dd59558b2c21d4b4212]) successfully deleted
I20260926 14:41:53.561280 30083 ts_tablet_manager.cc:1905] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:41:53.563797 30083 ts_tablet_manager.cc:1918] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:41:53.563884 30083 log.cc:1196] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/972b77e357484c73afc5273843bd1216
I20260926 14:41:53.564172 30083 ts_tablet_manager.cc:1939] T 972b77e357484c73afc5273843bd1216 P 849d008645cb4a77b6760e183893be58: Deleting consensus metadata
I20260926 14:41:53.565359 29541 catalog_manager.cc:4841] TS 849d008645cb4a77b6760e183893be58 (127.16.90.195:34725): tablet 972b77e357484c73afc5273843bd1216 (table test-table [id=b5c29acb108c4dd59558b2c21d4b4212]) successfully deleted
W20260926 14:41:53.567380 29540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:35437 (error 108)
I20260926 14:41:53.568562 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 29759
I20260926 14:41:53.571460 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:53.571135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:35437 (local address 127.0.0.1:49826)
0926 14:41:53.571307 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:53.571373 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260926 14:41:53.571631 29540 catalog_manager.cc:4584] TS a9c67381f0b742029365346f5b6e0d55 (127.16.90.193:35437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 972b77e357484c73afc5273843bd1216: Network error: Client connection negotiation failed: client connection to 127.16.90.193:35437: connect: Connection refused (error 111)
I20260926 14:41:53.576309 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 29894
I20260926 14:41:53.589056 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 29523
I20260926 14:41:53.595038 16747 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 14:41:53.929278 16747 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 14:41:53.929396 16747 mini_ranger.cc:66] Stopping Ranger...
I20260926 14:41:54.263067 16747 mini_ranger.cc:68] Stopped Ranger
2026-09-26 14:41:54.263 UTC [28587] LOG: received smart shutdown request
2026-09-26 14:41:54.266 UTC [28587] LOG: background worker "logical replication launcher" (PID 28596) exited with exit code 1
2026-09-26 14:41:54.267 UTC [28591] LOG: shutting down
2026-09-26 14:41:54.398 UTC [28587] LOG: database system is shut down
2026-09-26T14:41:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46504 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:41:54 dist-test-slave-x4sn krb5kdc[30094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:41:54 dist-test-slave-x4sn krb5kdc[30094](info): set up 2 sockets
Sep 26 14:41:54 dist-test-slave-x4sn krb5kdc[30094](info): commencing operation
krb5kdc: starting...
W20260926 14:41:55.453066 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:41:55 dist-test-slave-x4sn krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:41:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:41:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:41:55.564163 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:39839
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:43441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:39839,127.16.90.253:43671,127.16.90.252:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:41:55.668391 30107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:55.668643 30107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:55.668682 30107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:55.672559 30107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:41:55.672618 30107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:55.672636 30107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:55.672662 30107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:41:55.672676 30107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:41:55.677758 30107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:43441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:39839,127.16.90.253:43671,127.16.90.252:44525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:39839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:55.678721 30107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:55.679654 30107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:55.687455 30115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:55.687628 30107 server_base.cc:1034] running on GCE node
W20260926 14:41:55.687453 30113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:55.687455 30112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:55.688309 30107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:55.689311 30107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:55.690467 30107 hybrid_clock.cc:648] HybridClock initialized: now 1790433715690452 us; error 26 us; skew 500 ppm
Sep 26 14:41:55 dist-test-slave-x4sn krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433715, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:55.693001 30107 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:41:55.693804 30107 webserver.cc:460] Webserver started at http://127.16.90.254:33253/ using document root <none> and password file <none>
I20260926 14:41:55.694295 30107 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:55.694357 30107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:41:55.694576 30107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:41:55.696547 30107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8"
format_stamp: "Formatted at 2026-09-26 14:41:55 on dist-test-slave-x4sn"
server_key: "124afb58e3fcca33fc02fe3063511316"
server_key_iv: "507694f2de5c91ffb6bdac7c201b358d"
server_key_version: "encryptionkey@0"
I20260926 14:41:55.697046 30107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8"
format_stamp: "Formatted at 2026-09-26 14:41:55 on dist-test-slave-x4sn"
server_key: "124afb58e3fcca33fc02fe3063511316"
server_key_iv: "507694f2de5c91ffb6bdac7c201b358d"
server_key_version: "encryptionkey@0"
I20260926 14:41:55.700407 30107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:41:55.702636 30122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:55.703882 30107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:41:55.703979 30107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8"
format_stamp: "Formatted at 2026-09-26 14:41:55 on dist-test-slave-x4sn"
server_key: "124afb58e3fcca33fc02fe3063511316"
server_key_iv: "507694f2de5c91ffb6bdac7c201b358d"
server_key_version: "encryptionkey@0"
I20260926 14:41:55.704067 30107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:55.722627 30107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:55.725626 30107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:55.725790 30107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:55.732354 30107 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:39839
I20260926 14:41:55.732420 30174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:39839 every 8 connection(s)
I20260926 14:41:55.733315 30107 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:41:55.736028 30175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:41:55.739151 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 30107
I20260926 14:41:55.739256 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:41:55.739490 16747 external_mini_cluster.cc:1428] Setting key 3860d172c9d6e019d628d41a497b393c
I20260926 14:41:55.739645 30175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.90.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.253@KRBTEST.COM" created.
Sep 26 14:41:55 dist-test-slave-x4sn krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433715, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:55.755194 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.741839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.16.90.254:39839)
0926 14:41:55.742252 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:55.742270 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:55.742456 (+ 186us) server_negotiation.cc:407] Connection header received
0926 14:41:55.743463 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:55.743469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:55.743740 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:55.743938 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:55.744670 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.746013 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.746945 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.747522 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.747777 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:55.750682 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:55.750687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:55.750697 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:55.750716 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:55.752382 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:55.752791 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:55.752796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:55.752798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:55.752896 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:55.753191 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:55.753195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:55.753197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:55.753686 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:55.753812 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:55.754201 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 14:41:55.754427 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":285,"spinlock_wait_cycles":17280,"thread_start_us":136,"threads_started":1}
I20260926 14:41:55.755200 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.741838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:39839 (local address 127.0.0.1:41244)
0926 14:41:55.742303 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.742351 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:55.743160 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:55.743931 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:55.743935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:55.744270 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:55.744529 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:55.744541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.746127 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.746129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:55.746832 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:55.746839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.747663 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.747691 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:55.747823 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:55.748350 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:55.748369 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:55.750551 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:55.752479 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:55.752487 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:55.752496 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:55.752698 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:55.752974 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:55.752977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:55.752979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:55.753078 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:55.753794 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:55.753801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:55.753958 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 14:41:55.754147 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 14:41:55.754496 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":59,"thread_start_us":110,"threads_started":1}
I20260926 14:41:55.757727 30175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:41:55.758365 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.758145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:43671 (local address 127.0.0.1:42620)
0926 14:41:55.758250 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.758315 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:43671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 14:41:55.758530 30125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.90.253:43671: connect: Connection refused (error 111)
I20260926 14:41:55.759346 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.759180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:43671 (local address 127.0.0.1:42622)
0926 14:41:55.759258 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.759300 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:43671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 14:41:55.759528 30175 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.253:43671: Network error: Client connection negotiation failed: client connection to 127.16.90.253:43671: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.253.keytab.
Entry for principal kudu/127.16.90.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.253.keytab.
I20260926 14:41:55.764093 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.253:43671
--webserver_interface=127.16.90.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.253.keytab
--principal=kudu/127.16.90.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:43441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:39839,127.16.90.253:43671,127.16.90.252:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:41:55.808733 30175 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } attempt: 1
I20260926 14:41:55.809584 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.809241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:43671 (local address 127.0.0.1:42624)
0926 14:41:55.809412 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.809493 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:43671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 14:41:55.810333 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.810077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:43671 (local address 127.0.0.1:42636)
0926 14:41:55.810211 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.810268 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:43671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:41:55.810459 30175 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.253:43671: Network error: Client connection negotiation failed: client connection to 127.16.90.253:43671: connect: Connection refused (error 111)
W20260926 14:41:55.868867 30182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:55.869129 30182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:55.869167 30182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:55.873423 30182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:41:55.873476 30182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:55.873494 30182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:55.873509 30182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:41:55.873522 30182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:41:55.878561 30182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:43441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:39839,127.16.90.253:43671,127.16.90.252:44525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.253.keytab
--principal=kudu/127.16.90.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.253:43671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.90.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:55.878626 30175 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } attempt: 2
I20260926 14:41:55.879415 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.879090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:43671 (local address 127.0.0.1:42648)
0926 14:41:55.879257 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.879328 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:43671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 14:41:55.879832 30182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:55.880070 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.879858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:43671 (local address 127.0.0.1:42656)
0926 14:41:55.879960 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.880008 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:43671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:41:55.880183 30175 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.253:43671: Network error: Client connection negotiation failed: client connection to 127.16.90.253:43671: connect: Connection refused (error 111)
I20260926 14:41:55.880885 30182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:55.888768 30190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:55.888768 30188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:55.888805 30187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:55.889058 30182 server_base.cc:1034] running on GCE node
I20260926 14:41:55.889626 30182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:55.890655 30182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:55.891809 30182 hybrid_clock.cc:648] HybridClock initialized: now 1790433715891796 us; error 23 us; skew 500 ppm
Sep 26 14:41:55 dist-test-slave-x4sn krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433715, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:55.894479 30182 init.cc:377] Logged in from keytab as kudu/127.16.90.253@KRBTEST.COM (short username kudu)
I20260926 14:41:55.895272 30182 webserver.cc:460] Webserver started at http://127.16.90.253:46139/ using document root <none> and password file <none>
I20260926 14:41:55.895812 30182 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:55.895881 30182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:41:55.896068 30182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:41:55.897959 30182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/data/instance:
uuid: "5be725286dff4c539186ffe326ab8f6a"
format_stamp: "Formatted at 2026-09-26 14:41:55 on dist-test-slave-x4sn"
server_key: "88ca2ef62e6f858b18978a5ec213ebba"
server_key_iv: "44bb076c903f11b4290009cd407ba537"
server_key_version: "encryptionkey@0"
I20260926 14:41:55.898485 30182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/wal/instance:
uuid: "5be725286dff4c539186ffe326ab8f6a"
format_stamp: "Formatted at 2026-09-26 14:41:55 on dist-test-slave-x4sn"
server_key: "88ca2ef62e6f858b18978a5ec213ebba"
server_key_iv: "44bb076c903f11b4290009cd407ba537"
server_key_version: "encryptionkey@0"
I20260926 14:41:55.901908 30182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 14:41:55.904114 30197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:55.905205 30182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:41:55.905318 30182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/wal
uuid: "5be725286dff4c539186ffe326ab8f6a"
format_stamp: "Formatted at 2026-09-26 14:41:55 on dist-test-slave-x4sn"
server_key: "88ca2ef62e6f858b18978a5ec213ebba"
server_key_iv: "44bb076c903f11b4290009cd407ba537"
server_key_version: "encryptionkey@0"
I20260926 14:41:55.905405 30182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:55.921262 30182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:55.924283 30182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:55.924455 30182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:55.930895 30182 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.253:43671
I20260926 14:41:55.930943 30249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.253:43671 every 8 connection(s)
I20260926 14:41:55.931816 30182 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
I20260926 14:41:55.934384 30250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:41:55.937966 30250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:41:55.938984 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 30182
I20260926 14:41:55.939090 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-1/wal/instance
I20260926 14:41:55.939342 16747 external_mini_cluster.cc:1428] Setting key a2e004dc0445afa132bda074e839c190
WARNING: no policy specified for kudu/127.16.90.252@KRBTEST.COM; defaulting to no policy
Sep 26 14:41:55 dist-test-slave-x4sn krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433715, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.253@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.252@KRBTEST.COM" created.
I20260926 14:41:55.950650 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.939777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.16.90.254:39839)
0926 14:41:55.939894 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:55.939898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:55.940845 (+ 947us) server_negotiation.cc:407] Connection header received
0926 14:41:55.941962 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:55.941965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:55.942018 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:55.942106 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:55.943248 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.943809 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.944892 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.945123 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.945210 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:55.947426 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:55.947429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:55.947431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:55.947455 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:55.948723 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:55.949243 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:55.949247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:55.949248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:55.949335 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:55.949610 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:55.949613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:55.949614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:55.949815 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:55.949905 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:55.950365 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 14:41:55.950492 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:41:55.951432 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.940256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:39839 (local address 127.0.0.1:41258)
0926 14:41:55.940701 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.940742 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:55.941723 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:55.942284 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:55.942293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:55.942621 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:55.943102 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:55.943115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.943919 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.943922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:55.944779 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:55.944788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.945258 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.945262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:55.945426 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:55.945935 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:55.945951 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:55.947277 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:55.948825 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:55.948831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:55.948844 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:55.949146 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:55.949419 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:55.949422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:55.949423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:55.949539 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:55.949893 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:55.949898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:55.950120 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 14:41:55.950280 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 14:41:55.950576 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260926 14:41:55.952783 30250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:41:55 dist-test-slave-x4sn krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433715, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.253@KRBTEST.COM for kudu/127.16.90.253@KRBTEST.COM
I20260926 14:41:55.960979 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.953238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:43671 (local address 127.0.0.1:42668)
0926 14:41:55.953343 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.953354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:55.953411 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:55.953925 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:55.953927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:55.954080 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:55.954149 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:55.954154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.955049 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.955052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:55.955574 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:55.955581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.956118 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.956121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:55.956216 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:55.956581 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:55.956593 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:55.958123 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:55.959697 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:55.959700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:55.959702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:55.959864 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:55.960103 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:55.960106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:55.960108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:55.960187 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:55.960560 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:55.960563 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:55.960640 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 14:41:55.960727 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 14:41:55.960838 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 14:41:55.961081 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.953377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.16.90.253:43671)
0926 14:41:55.953622 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:55.953630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:55.953646 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:41:55.953683 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:55.953687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:55.953819 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:55.953928 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:55.954247 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.954951 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.955689 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.956023 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.956112 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:55.958232 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:55.958240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:55.958250 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:55.958268 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:55.959583 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:55.959954 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:55.959958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:55.959959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:55.960026 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:55.960258 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:55.960262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:55.960264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:55.960469 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:55.960576 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:55.960810 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 14:41:55.960923 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
I20260926 14:41:55.962785 30250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.252.keytab.
Entry for principal kudu/127.16.90.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.252.keytab.
I20260926 14:41:55.963435 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.963191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53702)
0926 14:41:55.963295 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.963371 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 14:41:55.963620 30200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
I20260926 14:41:55.963740 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.252:44525
--webserver_interface=127.16.90.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.252.keytab
--principal=kudu/127.16.90.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:43441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:39839,127.16.90.253:43671,127.16.90.252:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:41:55.964468 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.964258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53712)
0926 14:41:55.964363 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.964412 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 14:41:55.964598 30250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:44525: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
I20260926 14:41:55.971318 30175 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } attempt: 3
Sep 26 14:41:55 dist-test-slave-x4sn krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433715, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.253@KRBTEST.COM
I20260926 14:41:55.979096 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.971775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:43671 (local address 127.0.0.1:42674)
0926 14:41:55.971902 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.971916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:55.972010 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:55.972223 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:55.972226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:55.972386 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:55.972487 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:55.972492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.973168 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.973172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:55.974086 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:55.974096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.974535 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.974538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:55.974639 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:55.975048 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:55.975062 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:55.976372 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:55.977809 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:55.977812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:55.977814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:55.977990 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:55.978236 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:55.978238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:55.978240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:55.978312 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:55.978631 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:55.978634 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:55.978729 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 14:41:55.978810 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 14:41:55.978936 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 14:41:55.979096 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.971796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.16.90.253:43671)
0926 14:41:55.971900 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:55.971904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:55.971970 (+ 66us) server_negotiation.cc:407] Connection header received
0926 14:41:55.972096 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:55.972099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:55.972138 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:55.972206 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:55.972588 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.973072 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.974200 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:55.974441 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:55.974522 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:55.976488 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:55.976492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:55.976494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:55.976513 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:55.977712 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:55.978073 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:55.978077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:55.978078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:55.978143 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:55.978388 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:55.978392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:55.978393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:55.978548 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:55.978626 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:55.978869 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 14:41:55.978963 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:41:55.980120 30175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:41:55.980743 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.980527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53716)
0926 14:41:55.980628 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.980685 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 14:41:55.981329 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:55.981109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53722)
0926 14:41:55.981230 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:55.981275 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 14:41:55.981429 30175 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:44525: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
I20260926 14:41:56.012563 30175 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } attempt: 1
I20260926 14:41:56.013401 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.013110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53728)
0926 14:41:56.013253 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.013317 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 14:41:56.013823 30250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } attempt: 1
I20260926 14:41:56.014160 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.013903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53734)
0926 14:41:56.014031 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.014090 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 14:41:56.014364 30175 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:44525: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
I20260926 14:41:56.014503 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.014188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53736)
0926 14:41:56.014375 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.014431 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 14:41:56.015151 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.014949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53740)
0926 14:41:56.015039 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.015089 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 14:41:56.015264 30250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:44525: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
W20260926 14:41:56.072355 30257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:56.072606 30257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:56.072643 30257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:56.076478 30257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:41:56.076529 30257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:56.076546 30257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:56.076562 30257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:41:56.076574 30257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:41:56.081645 30257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:43441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:39839,127.16.90.253:43671,127.16.90.252:44525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.252.keytab
--principal=kudu/127.16.90.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.252:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.90.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:56.082605 30257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:56.083426 30250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } attempt: 2
I20260926 14:41:56.083580 30257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:41:56.084244 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.083932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53746)
0926 14:41:56.084087 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.084158 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 14:41:56.084905 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.084713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53760)
0926 14:41:56.084808 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.084853 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 14:41:56.085035 30250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:44525: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
I20260926 14:41:56.089516 30175 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } attempt: 2
I20260926 14:41:56.090178 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.089892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53776)
0926 14:41:56.090034 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.090098 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 14:41:56.090927 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.090672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53790)
0926 14:41:56.090815 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.090863 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260926 14:41:56.091120 30175 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:44525: Network error: Client connection negotiation failed: client connection to 127.16.90.252:44525: connect: Connection refused (error 111)
W20260926 14:41:56.091881 30263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:56.091895 30262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:56.091905 30265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:56.092188 30257 server_base.cc:1034] running on GCE node
I20260926 14:41:56.092760 30257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:56.093806 30257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:56.094955 30257 hybrid_clock.cc:648] HybridClock initialized: now 1790433716094940 us; error 25 us; skew 500 ppm
Sep 26 14:41:56 dist-test-slave-x4sn krb5kdc[30094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433716, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:56.097491 30257 init.cc:377] Logged in from keytab as kudu/127.16.90.252@KRBTEST.COM (short username kudu)
I20260926 14:41:56.098289 30257 webserver.cc:460] Webserver started at http://127.16.90.252:41955/ using document root <none> and password file <none>
I20260926 14:41:56.098788 30257 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:56.098851 30257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:41:56.099023 30257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:41:56.100980 30257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/data/instance:
uuid: "162d1412c67e480aac9994d986bd6eee"
format_stamp: "Formatted at 2026-09-26 14:41:56 on dist-test-slave-x4sn"
server_key: "9c919333284fa002c6f56ae4aad5a599"
server_key_iv: "475c166a11319fbf69b33dc3755dfe3c"
server_key_version: "encryptionkey@0"
I20260926 14:41:56.101487 30257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/wal/instance:
uuid: "162d1412c67e480aac9994d986bd6eee"
format_stamp: "Formatted at 2026-09-26 14:41:56 on dist-test-slave-x4sn"
server_key: "9c919333284fa002c6f56ae4aad5a599"
server_key_iv: "475c166a11319fbf69b33dc3755dfe3c"
server_key_version: "encryptionkey@0"
I20260926 14:41:56.104926 30257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:41:56.107174 30272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:56.108408 30257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:41:56.108536 30257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/wal
uuid: "162d1412c67e480aac9994d986bd6eee"
format_stamp: "Formatted at 2026-09-26 14:41:56 on dist-test-slave-x4sn"
server_key: "9c919333284fa002c6f56ae4aad5a599"
server_key_iv: "475c166a11319fbf69b33dc3755dfe3c"
server_key_version: "encryptionkey@0"
I20260926 14:41:56.108628 30257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:56.121308 30257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:56.125054 30257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:56.125319 30257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:56.132138 30257 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.252:44525
I20260926 14:41:56.132153 30325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.252:44525 every 8 connection(s)
I20260926 14:41:56.133038 30257 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
I20260926 14:41:56.135617 30326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:41:56.139390 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 30257
I20260926 14:41:56.139492 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433562403030-16747-0/minicluster-data/master-2/wal/instance
I20260926 14:41:56.139249 30326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:41:56.139786 16747 external_mini_cluster.cc:1428] Setting key b6bbb91902658a28ecdf40ce80ff8fb3
Sep 26 14:41:56 dist-test-slave-x4sn krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433716, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.252@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
Sep 26 14:41:56 dist-test-slave-x4sn krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.252@KRBTEST.COM
I20260926 14:41:56.152562 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.141757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.16.90.254:39839)
0926 14:41:56.141883 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:56.141887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:56.142116 (+ 229us) server_negotiation.cc:407] Connection header received
0926 14:41:56.142505 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:56.142508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:56.142559 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:56.142637 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:56.143630 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.144171 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.145521 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.145855 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.145963 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.148643 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:56.148646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:56.148648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:56.148679 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:56.150205 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.150651 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.150654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.150656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.150747 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.151336 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.151339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.151341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.151559 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:56.151688 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:56.152129 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 14:41:56.152246 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 14:41:56.153232 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.141637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:39839 (local address 127.0.0.1:41268)
0926 14:41:56.141971 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.142019 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:56.142333 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:56.142713 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:56.142718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:56.143151 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:56.143494 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:56.143509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.144271 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.144274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:56.145408 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:56.145418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.145985 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.145987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:56.146099 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.146777 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:56.146796 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:56.148509 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:56.150304 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:56.150311 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:56.150318 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:56.150557 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:56.151098 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:56.151101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:56.151102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:56.151248 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:56.151695 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:56.151702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:56.151911 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 14:41:56.152073 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 14:41:56.152385 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":57,"spinlock_wait_cycles":768,"thread_start_us":88,"threads_started":1}
I20260926 14:41:56.153239 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.141073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.16.90.252:44525)
0926 14:41:56.141446 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:56.141459 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:56.141491 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:41:56.142245 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:56.142264 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:56.142468 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:56.142662 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:56.143493 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.144371 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.145481 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.146045 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.146133 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.149304 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:56.149311 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:56.149324 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:56.149354 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:56.150749 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.151082 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.151099 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.151100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.151245 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.151492 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.151495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.151497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.151908 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:56.152040 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:56.152278 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 14:41:56.152437 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":247,"spinlock_wait_cycles":3968,"thread_start_us":98,"threads_started":1}
I20260926 14:41:56.154654 30326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:41:56 dist-test-slave-x4sn krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433716, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.252@KRBTEST.COM for kudu/127.16.90.253@KRBTEST.COM
Sep 26 14:41:56 dist-test-slave-x4sn krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:56.162891 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.155227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:43671 (local address 127.0.0.1:42688)
0926 14:41:56.155370 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.155380 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:56.155437 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:56.155647 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:56.155650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:56.155832 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:56.155947 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:56.155954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.156866 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.156869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:56.157588 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:56.157596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.158103 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.158105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:56.158185 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.158573 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:56.158587 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:56.159747 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:56.161472 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:56.161476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:56.161478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:56.161693 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:56.161959 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:56.161961 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:56.161962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:56.162028 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:56.162415 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:56.162418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:56.162506 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 14:41:56.162591 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:41:56.162727 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 14:41:56.162891 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.155327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.16.90.253:43671)
0926 14:41:56.155436 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:56.155441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:56.155455 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:41:56.155498 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:56.155501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:56.155548 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:56.155626 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:56.156054 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.156777 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.157683 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.158016 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.158121 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.159874 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:56.159877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:56.159879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:56.159904 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:56.161358 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.161788 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.161792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.161794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.161879 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.162098 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.162101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.162103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.162329 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:56.162409 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:56.162652 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 14:41:56.162755 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:41:56.163533 30326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:41:56.163815 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.155634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41280 (local address 127.16.90.254:39839)
0926 14:41:56.155774 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:56.155778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:56.155825 (+ 47us) server_negotiation.cc:407] Connection header received
0926 14:41:56.155905 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:56.155908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:56.155945 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:56.156018 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:56.156696 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.157357 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.158292 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.158532 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.158628 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.160783 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:56.160787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:56.160789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:56.160815 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:56.162168 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.162511 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.162515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.162517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.162594 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.162820 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.162823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.162825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.163020 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:56.163251 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:56.163547 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 14:41:56.163658 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 14:41:56 dist-test-slave-x4sn krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433716, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.252@KRBTEST.COM for kudu/127.16.90.252@KRBTEST.COM
Sep 26 14:41:56 dist-test-slave-x4sn krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.253@KRBTEST.COM
I20260926 14:41:56.170790 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.164063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53806)
0926 14:41:56.164181 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.164193 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:56.164263 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:56.164439 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:56.164440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:56.164585 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:56.164671 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:56.164677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.165448 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.165451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:56.165938 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:56.165945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.166374 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.166378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:56.166450 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.166811 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:56.166821 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:56.168021 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:56.169528 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:56.169531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:56.169533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:56.169731 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:56.169985 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:56.169988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:56.169990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:56.170069 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:56.170436 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:56.170438 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:56.170510 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 14:41:56.170588 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 14:41:56.170680 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 14:41:56.170936 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.164191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.16.90.252:44525)
0926 14:41:56.164276 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:56.164280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:56.164292 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:41:56.164323 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:56.164325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:56.164360 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:56.164427 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:56.164768 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.165355 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.166032 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.166289 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.166369 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.168155 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:56.168158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:56.168160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:56.168179 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:56.169434 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.169822 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.169825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.169827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.169897 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.170149 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.170152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.170154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.170348 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:56.170444 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:56.170661 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 14:41:56.170763 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":31}
I20260926 14:41:56.172747 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.165041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.16.90.253:43671)
0926 14:41:56.165134 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:56.165138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:56.165149 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:41:56.165236 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:56.165239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:56.165275 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:56.165335 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:56.165870 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.166331 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.167022 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.167277 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.167368 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.169613 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:56.169616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:56.169618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:56.169637 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:56.171304 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.171640 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.171643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.171645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.171750 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.171985 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.171989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.171991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.172205 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:56.172434 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:56.172505 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 14:41:56.172612 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:41:56.174190 30326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee: Bootstrap starting.
I20260926 14:41:56.175736 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:41:56.176179 30250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } attempt: 3
I20260926 14:41:56.176952 30326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee: Neither blocks nor log segments found. Creating new log.
I20260926 14:41:56.177662 30326 log.cc:824] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee: Log is configured to *not* fsync() on all Append() calls
I20260926 14:41:56.180104 30326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee: No bootstrap required, opened a new log
Sep 26 14:41:56 dist-test-slave-x4sn krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433715, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.253@KRBTEST.COM for kudu/127.16.90.252@KRBTEST.COM
I20260926 14:41:56.185621 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.176525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53810)
0926 14:41:56.176624 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.176638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:56.176705 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:56.176954 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:56.176958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:56.177151 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:56.177251 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:56.177258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.178241 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.178244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:56.178991 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:56.179000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.179570 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.179574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:56.179649 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.180782 (+ 1133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:56.180796 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:56.182222 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:56.184050 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:56.184053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:56.184055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:56.184297 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:56.184574 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:56.184577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:56.184579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:56.184677 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:56.185162 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:56.185164 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:56.185236 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 14:41:56.185315 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 14:41:56.185456 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 14:41:56.185585 30326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } }
I20260926 14:41:56.185937 30326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:41:56.186029 30326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 162d1412c67e480aac9994d986bd6eee, State: Initialized, Role: FOLLOWER
I20260926 14:41:56.186188 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.176603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.16.90.252:44525)
0926 14:41:56.176699 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:56.176703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:56.176716 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:41:56.176755 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:56.176757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:56.176811 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:56.177361 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:56.177446 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.178118 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.179179 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.179469 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.179557 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.182358 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:56.182361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:56.182364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:56.182386 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:56.183928 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.184406 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.184409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.184411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.184491 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.184769 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.184773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.184774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.185014 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:56.185118 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:56.185911 (+ 793us) server_negotiation.cc:299] Negotiation successful
0926 14:41:56.186013 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:41:56.186597 30326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } }
I20260926 14:41:56.187887 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.178344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.16.90.253:43671)
0926 14:41:56.178438 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:56.178441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:56.179877 (+ 1436us) server_negotiation.cc:407] Connection header received
0926 14:41:56.180028 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:56.180031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:56.180069 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:56.180114 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:56.180980 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.181666 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.182701 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.183015 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.183112 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.184451 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:56.184454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:56.184457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:56.184478 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:56.186022 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.186821 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.186824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.186826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.186906 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.187159 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.187163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.187164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.187319 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:56.187444 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:56.187642 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 14:41:56.187756 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:41:56.187985 30326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:41:56.189221 30250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a: Bootstrap starting.
I20260926 14:41:56.190286 30175 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } attempt: 3
I20260926 14:41:56.192381 30250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a: Neither blocks nor log segments found. Creating new log.
W20260926 14:41:56.195830 30351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:41:56.195950 30351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 14:41:56 dist-test-slave-x4sn krb5kdc[30094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433715, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.252@KRBTEST.COM
I20260926 14:41:56.191954 30257 master_runner.cc:186] Error getting master registration for 127.16.90.254:39839: OK, instance_id { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" instance_seqno: 1790433715730567 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:41:56.196457 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.177939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.16.90.254:39839)
0926 14:41:56.178055 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:56.178059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:56.178900 (+ 841us) server_negotiation.cc:407] Connection header received
0926 14:41:56.178990 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:56.178993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:56.179042 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:56.179124 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:56.179904 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.180622 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.181698 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.182015 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.182109 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.184253 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:56.184257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:56.184259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:56.184285 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:56.185826 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.186699 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.186703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.186704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.186789 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.192775 (+ 5986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.192779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.192781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.192994 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:56.193116 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:56.193737 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 14:41:56.193863 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:41:56.196523 30257 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:41:56.197536 30250 log.cc:824] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:41:56.198135 30340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } } }
I20260926 14:41:56.198261 30340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:41:56.199764 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.190648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:44525 (local address 127.0.0.1:53828)
0926 14:41:56.190864 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 14:41:56.190877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:41:56.190946 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:41:56.192011 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:41:56.192014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:41:56.192182 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:41:56.192277 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:56.192283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.193240 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.193244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:56.193973 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:41:56.193984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.194645 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.194649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:41:56.194729 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.195117 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:41:56.195129 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:41:56.196531 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:41:56.198222 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:56.198225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:56.198228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:56.198447 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:56.198759 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:41:56.198762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:41:56.198764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:41:56.198859 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:41:56.199276 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:41:56.199279 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:41:56.199370 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 14:41:56.199444 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 14:41:56.199571 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260926 14:41:56.200222 30250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a: No bootstrap required, opened a new log
I20260926 14:41:56.200353 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.191066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.16.90.252:44525)
0926 14:41:56.191791 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:56.191795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:56.191809 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:41:56.191841 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:56.191843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:56.191880 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:56.192382 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:56.192452 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.193109 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.194146 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.194432 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.194550 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.196675 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:56.196678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:56.196681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:56.196703 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:56.198111 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.198567 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.198570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.198572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.198652 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.198959 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.198962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.198964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.199162 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:56.200023 (+ 861us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:56.200110 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 14:41:56.200216 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656}
I20260926 14:41:56.201201 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:56.179266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.16.90.252:44525)
0926 14:41:56.180017 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:56.180025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:56.188121 (+ 8096us) server_negotiation.cc:407] Connection header received
0926 14:41:56.188158 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:56.188160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:56.188200 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:56.188276 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:56.188775 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.189429 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.190605 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:56.190918 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:56.191002 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:56.192433 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:56.192437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:56.192439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:56.192460 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:56.198940 (+ 6480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.199389 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.199392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.199394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.199492 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:56.200124 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:56.200127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:56.200128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:56.200347 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:56.200460 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:56.200924 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 14:41:56.201033 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":95,"threads_started":1}
I20260926 14:41:56.203402 30175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8: Bootstrap starting.
I20260926 14:41:56.204028 30250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } }
I20260926 14:41:56.204216 30250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:41:56.204301 30250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5be725286dff4c539186ffe326ab8f6a, State: Initialized, Role: FOLLOWER
I20260926 14:41:56.204730 30250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } }
I20260926 14:41:56.206043 30250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:41:56.206172 30175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8: Neither blocks nor log segments found. Creating new log.
I20260926 14:41:56.207111 30175 log.cc:824] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8: Log is configured to *not* fsync() on all Append() calls
I20260926 14:41:56.207274 30354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } } }
I20260926 14:41:56.207450 30354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:41:56.209276 30175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8: No bootstrap required, opened a new log
W20260926 14:41:56.211153 30366 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 14:41:56.213524 30175 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } }
I20260926 14:41:56.213759 30175 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:41:56.213814 30175 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc40a294872a4cd5bfb3ca4a9fc386b8, State: Initialized, Role: FOLLOWER
I20260926 14:41:56.214278 30175 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } }
I20260926 14:41:56.215567 30175 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 14:41:56.220909 30379 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:41:56.220994 30379 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:41:56.223161 30368 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } } }
I20260926 14:41:56.223282 30368 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:41:56.226281 30107 master_runner.cc:263] No leader master found from master fc40a294872a4cd5bfb3ca4a9fc386b8
I20260926 14:41:56.226348 30107 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:41:56.214829 30182 master_runner.cc:186] Error getting master registration for 127.16.90.254:39839: OK, instance_id { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" instance_seqno: 1790433715730567 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:41:56.226923 30182 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 14:41:56.227010 30366 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 14:41:56.227352 30128 tablet.cc:2369] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:41:56.376964 30278 tablet.cc:2369] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:41:56.378769 30340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:41:56.378957 30340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } }
I20260926 14:41:56.380004 30340 leader_election.cc:288] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc40a294872a4cd5bfb3ca4a9fc386b8 (127.16.90.254:39839), 5be725286dff4c539186ffe326ab8f6a (127.16.90.253:43671)
I20260926 14:41:56.380661 30224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "162d1412c67e480aac9994d986bd6eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5be725286dff4c539186ffe326ab8f6a" is_pre_election: true
I20260926 14:41:56.380867 30147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "162d1412c67e480aac9994d986bd6eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" is_pre_election: true
I20260926 14:41:56.380998 30224 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 162d1412c67e480aac9994d986bd6eee in term 0.
I20260926 14:41:56.381136 30147 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 162d1412c67e480aac9994d986bd6eee in term 0.
I20260926 14:41:56.381515 30275 leader_election.cc:302] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 162d1412c67e480aac9994d986bd6eee, fc40a294872a4cd5bfb3ca4a9fc386b8; no voters:
I20260926 14:41:56.381799 30340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:41:56.381880 30340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:41:56.381920 30340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:41:56.382941 30340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } }
I20260926 14:41:56.383244 30340 leader_election.cc:288] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [CANDIDATE]: Term 1 election: Requested vote from peers fc40a294872a4cd5bfb3ca4a9fc386b8 (127.16.90.254:39839), 5be725286dff4c539186ffe326ab8f6a (127.16.90.253:43671)
I20260926 14:41:56.383647 30147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "162d1412c67e480aac9994d986bd6eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8"
I20260926 14:41:56.383622 30224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "162d1412c67e480aac9994d986bd6eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5be725286dff4c539186ffe326ab8f6a"
I20260926 14:41:56.383796 30147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:41:56.383797 30224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:41:56.385035 30224 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 162d1412c67e480aac9994d986bd6eee in term 1.
I20260926 14:41:56.385035 30147 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 162d1412c67e480aac9994d986bd6eee in term 1.
I20260926 14:41:56.385353 30275 leader_election.cc:302] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 162d1412c67e480aac9994d986bd6eee, 5be725286dff4c539186ffe326ab8f6a; no voters:
I20260926 14:41:56.385613 30340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:41:56.385839 30340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [term 1 LEADER]: Becoming Leader. State: Replica: 162d1412c67e480aac9994d986bd6eee, State: Running, Role: LEADER
I20260926 14:41:56.386178 30340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } }
I20260926 14:41:56.388111 30381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 162d1412c67e480aac9994d986bd6eee. Latest consensus state: current_term: 1 leader_uuid: "162d1412c67e480aac9994d986bd6eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } } }
I20260926 14:41:56.388229 30381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [sys.catalog]: This master's current role is: LEADER
I20260926 14:41:56.388593 30383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:41:56.389496 30383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:41:56.393867 30147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [term 1 FOLLOWER]: Refusing update from remote peer 162d1412c67e480aac9994d986bd6eee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:41:56.393867 30224 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [term 1 FOLLOWER]: Refusing update from remote peer 162d1412c67e480aac9994d986bd6eee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:41:56.394511 30381 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [LEADER]: Connected to new peer: Peer: permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:41:56.394683 30340 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:41:56.398756 30368 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 162d1412c67e480aac9994d986bd6eee. Latest consensus state: current_term: 1 leader_uuid: "162d1412c67e480aac9994d986bd6eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } } }
I20260926 14:41:56.398890 30368 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:41:56.399035 30354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 162d1412c67e480aac9994d986bd6eee. Latest consensus state: current_term: 1 leader_uuid: "162d1412c67e480aac9994d986bd6eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } } }
I20260926 14:41:56.399140 30354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:41:56.402235 30340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "162d1412c67e480aac9994d986bd6eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } } }
I20260926 14:41:56.402362 30340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [sys.catalog]: This master's current role is: LEADER
I20260926 14:41:56.402889 30384 mvcc.cc:205] Tried to move back new op lower bound from 7333616502343053312 to 7333616502320406528. Current Snapshot: MvccSnapshot[applied={T|T < 7333616502343053312}]
I20260926 14:41:56.403466 30368 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "162d1412c67e480aac9994d986bd6eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } } }
I20260926 14:41:56.403597 30368 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:41:56.404008 30385 mvcc.cc:205] Tried to move back new op lower bound from 7333616502343053312 to 7333616502320406528. Current Snapshot: MvccSnapshot[applied={T|T < 7333616502343053312}]
I20260926 14:41:56.405104 30381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "162d1412c67e480aac9994d986bd6eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } } }
I20260926 14:41:56.405217 30381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee [sys.catalog]: This master's current role is: LEADER
I20260926 14:41:56.406134 30354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "162d1412c67e480aac9994d986bd6eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc40a294872a4cd5bfb3ca4a9fc386b8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39839 } } peers { permanent_uuid: "5be725286dff4c539186ffe326ab8f6a" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 43671 } } peers { permanent_uuid: "162d1412c67e480aac9994d986bd6eee" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 44525 } } }
I20260926 14:41:56.406251 30354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:41:56.407596 30383 catalog_manager.cc:1347] Generated new cluster ID: 2bdbb3c7293a4127b66f68295c2dca75
I20260926 14:41:56.407653 30383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:41:56.426808 30383 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:41:56.428061 30383 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:41:56.443456 30383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 162d1412c67e480aac9994d986bd6eee: Generated new TSK 0
I20260926 14:41:56.444362 30383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:41:57.222379 30379 catalog_manager.cc:1259] Loaded cluster ID: 2bdbb3c7293a4127b66f68295c2dca75
I20260926 14:41:57.222469 30379 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8: loading cluster ID for follower catalog manager: success
I20260926 14:41:57.224181 30379 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8: acquiring CA information for follower catalog manager: success
I20260926 14:41:57.225317 30379 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc40a294872a4cd5bfb3ca4a9fc386b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:41:57.228171 30366 catalog_manager.cc:1259] Loaded cluster ID: 2bdbb3c7293a4127b66f68295c2dca75
I20260926 14:41:57.228230 30366 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a: loading cluster ID for follower catalog manager: success
I20260926 14:41:57.229746 30366 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a: acquiring CA information for follower catalog manager: success
I20260926 14:41:57.230794 30366 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5be725286dff4c539186ffe326ab8f6a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:41:57.511351 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 30107
I20260926 14:41:57.516964 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 30182
I20260926 14:41:57.521900 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 30257
2026-09-26T14:41:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3017 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:41:57 dist-test-slave-x4sn krb5kdc[30399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:41:57 dist-test-slave-x4sn krb5kdc[30399](info): set up 2 sockets
Sep 26 14:41:57 dist-test-slave-x4sn krb5kdc[30399](info): commencing operation
krb5kdc: starting...
W20260926 14:41:58.473600 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:41:58 dist-test-slave-x4sn krb5kdc[30399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:41:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:41:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:41:58.588173 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:45611
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:45611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:41:58.692984 30412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:58.693224 30412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:58.693265 30412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:58.697214 30412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:41:58.697268 30412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:58.697288 30412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:58.697319 30412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:41:58.697343 30412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:41:58.702376 30412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:45611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:45611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:58.703325 30412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:41:58.704322 30412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:41:58.712184 30420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:58.712297 30412 server_base.cc:1034] running on GCE node
W20260926 14:41:58.712146 30417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:41:58.712147 30418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:41:58.713001 30412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:41:58.713995 30412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:41:58.715147 30412 hybrid_clock.cc:648] HybridClock initialized: now 1790433718715134 us; error 27 us; skew 500 ppm
Sep 26 14:41:58 dist-test-slave-x4sn krb5kdc[30399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:41:58.717710 30412 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:41:58.718489 30412 webserver.cc:460] Webserver started at http://127.16.90.254:44985/ using document root <none> and password file <none>
I20260926 14:41:58.719003 30412 fs_manager.cc:359] Metadata directory not provided
I20260926 14:41:58.719058 30412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:41:58.719219 30412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:41:58.721225 30412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "23c1362220c8473daf80da4e24968237"
format_stamp: "Formatted at 2026-09-26 14:41:58 on dist-test-slave-x4sn"
server_key: "087724870901067d8ad2a7855b3b1a0e"
server_key_iv: "e4ad6b08018abe1eb848eb8dea520f85"
server_key_version: "encryptionkey@0"
I20260926 14:41:58.721725 30412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "23c1362220c8473daf80da4e24968237"
format_stamp: "Formatted at 2026-09-26 14:41:58 on dist-test-slave-x4sn"
server_key: "087724870901067d8ad2a7855b3b1a0e"
server_key_iv: "e4ad6b08018abe1eb848eb8dea520f85"
server_key_version: "encryptionkey@0"
I20260926 14:41:58.725153 30412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:41:58.727411 30427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:41:58.728561 30412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:41:58.728654 30412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "23c1362220c8473daf80da4e24968237"
format_stamp: "Formatted at 2026-09-26 14:41:58 on dist-test-slave-x4sn"
server_key: "087724870901067d8ad2a7855b3b1a0e"
server_key_iv: "e4ad6b08018abe1eb848eb8dea520f85"
server_key_version: "encryptionkey@0"
I20260926 14:41:58.728749 30412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:41:58.739456 30412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:41:58.742410 30412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:41:58.742573 30412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:41:58.749130 30412 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:45611
I20260926 14:41:58.749141 30479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:45611 every 8 connection(s)
I20260926 14:41:58.750005 30412 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:41:58.752635 30480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:41:58.752851 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 30412
I20260926 14:41:58.752960 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:41:58.753230 16747 external_mini_cluster.cc:1428] Setting key 225d0ead232b2c57a0f88daf71113024
I20260926 14:41:58.758805 30480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237: Bootstrap starting.
I20260926 14:41:58.761327 30480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237: Neither blocks nor log segments found. Creating new log.
Sep 26 14:41:58 dist-test-slave-x4sn krb5kdc[30399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:41:58.762042 30480 log.cc:824] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237: Log is configured to *not* fsync() on all Append() calls
I20260926 14:41:58.763737 30480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237: No bootstrap required, opened a new log
I20260926 14:41:58.766127 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:41:58.754351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45284 (local address 127.16.90.254:45611)
0926 14:41:58.754731 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 14:41:58.754746 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:41:58.754780 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:41:58.755446 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:41:58.755462 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:41:58.755814 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:41:58.756078 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:41:58.756920 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:58.757861 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:58.758980 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:41:58.759436 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:41:58.759546 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:41:58.762085 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:41:58.762090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:41:58.762101 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:41:58.762126 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:41:58.763607 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:58.764053 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:58.764058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:58.764062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:58.764165 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:41:58.764408 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:41:58.764411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:41:58.764413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:41:58.764774 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:41:58.764950 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 14:41:58.765205 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 14:41:58.765464 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260926 14:41:58.767133 30480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c1362220c8473daf80da4e24968237" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45611 } }
I20260926 14:41:58.767385 30480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:41:58.767436 30480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c1362220c8473daf80da4e24968237, State: Initialized, Role: FOLLOWER
I20260926 14:41:58.767895 30480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c1362220c8473daf80da4e24968237" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45611 } }
I20260926 14:41:58.768031 30480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:41:58.768127 30480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:41:58.768234 30480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:41:58.768808 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:41:58.769142 30480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c1362220c8473daf80da4e24968237" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45611 } }
I20260926 14:41:58.769436 30480 leader_election.cc:302] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23c1362220c8473daf80da4e24968237; no voters:
I20260926 14:41:58.769732 30480 leader_election.cc:288] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:41:58.769807 30485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:41:58.770015 30485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [term 1 LEADER]: Becoming Leader. State: Replica: 23c1362220c8473daf80da4e24968237, State: Running, Role: LEADER
I20260926 14:41:58.770352 30485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c1362220c8473daf80da4e24968237" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45611 } }
I20260926 14:41:58.770962 30480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:41:58.771473 30485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23c1362220c8473daf80da4e24968237. Latest consensus state: current_term: 1 leader_uuid: "23c1362220c8473daf80da4e24968237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c1362220c8473daf80da4e24968237" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45611 } } }
I20260926 14:41:58.771648 30485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [sys.catalog]: This master's current role is: LEADER
I20260926 14:41:58.771898 30486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23c1362220c8473daf80da4e24968237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c1362220c8473daf80da4e24968237" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45611 } } }
I20260926 14:41:58.771991 30486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237 [sys.catalog]: This master's current role is: LEADER
I20260926 14:41:58.772131 30492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:41:58.774740 30492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:41:58.779886 30492 catalog_manager.cc:1347] Generated new cluster ID: ea1cb0183341453b88a6915bf0a15698
I20260926 14:41:58.779953 30492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:41:58.797541 30492 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:41:58.798444 30492 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:41:58.807308 30492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23c1362220c8473daf80da4e24968237: Generated new TSK 0
I20260926 14:41:58.807986 30492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:41:58.824499 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 30412
2026-09-26T14:41:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1301 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:41:58 dist-test-slave-x4sn krb5kdc[30507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:41:58 dist-test-slave-x4sn krb5kdc[30507](info): set up 2 sockets
Sep 26 14:41:58 dist-test-slave-x4sn krb5kdc[30507](info): commencing operation
krb5kdc: starting...
W20260926 14:41:59.773149 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:41:59 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:41:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:41:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:41:59.883984 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:37775
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:37775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:41:59.988788 30520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:41:59.989051 30520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:41:59.989090 30520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:41:59.992972 30520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:41:59.993027 30520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:41:59.993045 30520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:41:59.993060 30520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:41:59.993076 30520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:41:59.998593 30520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:37775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:37775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:41:59.999577 30520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:00.000644 30520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:00.008769 30526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:00.008790 30528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:00.008869 30520 server_base.cc:1034] running on GCE node
W20260926 14:42:00.008775 30525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:00.009581 30520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:00.010648 30520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:00.011791 30520 hybrid_clock.cc:648] HybridClock initialized: now 1790433720011767 us; error 38 us; skew 500 ppm
Sep 26 14:42:00 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:00.019536 30520 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:42:00.020527 30520 webserver.cc:460] Webserver started at http://127.16.90.254:44663/ using document root <none> and password file <none>
I20260926 14:42:00.021078 30520 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:00.021148 30520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:00.021358 30520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:42:00.023487 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "0fa183633f3540299267b4e1071c1831"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "a32cb0c7f7b0a615d15a889e4fa2e98c"
server_key_iv: "26d150adbc38b8a3dd840b0bebf4a76d"
server_key_version: "encryptionkey@0"
I20260926 14:42:00.024065 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "0fa183633f3540299267b4e1071c1831"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "a32cb0c7f7b0a615d15a889e4fa2e98c"
server_key_iv: "26d150adbc38b8a3dd840b0bebf4a76d"
server_key_version: "encryptionkey@0"
I20260926 14:42:00.027647 30520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:42:00.030159 30535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:00.031354 30520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:42:00.031473 30520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "0fa183633f3540299267b4e1071c1831"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "a32cb0c7f7b0a615d15a889e4fa2e98c"
server_key_iv: "26d150adbc38b8a3dd840b0bebf4a76d"
server_key_version: "encryptionkey@0"
I20260926 14:42:00.031567 30520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:00.045797 30520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:00.049078 30520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:00.049265 30520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:00.056291 30520 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:37775
I20260926 14:42:00.056527 30587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:37775 every 8 connection(s)
I20260926 14:42:00.057279 30520 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:42:00.059047 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 30520
I20260926 14:42:00.059161 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:42:00.059413 16747 external_mini_cluster.cc:1428] Setting key 89069aeddd9a8c3ffb70a2b46588c3a6
I20260926 14:42:00.060187 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:00.066854 30588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Bootstrap starting.
Sep 26 14:42:00 dist-test-slave-x4sn krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:00.069311 30588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:00.070225 30588 log.cc:824] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:00.072367 30588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: No bootstrap required, opened a new log
I20260926 14:42:00.073952 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.060846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.16.90.254:37775)
0926 14:42:00.061269 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:00.061287 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:00.061331 (+ 44us) server_negotiation.cc:407] Connection header received
0926 14:42:00.062150 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:00.062181 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:00.062599 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:00.062963 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:00.063623 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.065079 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.065927 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.066580 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.066743 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.069130 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:00.069137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:00.069154 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:00.069195 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:00.070893 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:00.071293 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:00.071299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:00.071304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:00.071433 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:00.071774 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:00.071777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:00.071779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:00.072272 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:00.072459 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:00.072742 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 14:42:00.073030 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":115,"threads_started":1}
I20260926 14:42:00.076570 30588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } }
I20260926 14:42:00.076790 30588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:00.076836 30588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa183633f3540299267b4e1071c1831, State: Initialized, Role: FOLLOWER
I20260926 14:42:00.077183 30588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } }
I20260926 14:42:00.077291 30588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:42:00.077334 30588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:42:00.077425 30588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:00.078450 30588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } }
I20260926 14:42:00.078745 30588 leader_election.cc:302] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fa183633f3540299267b4e1071c1831; no voters:
I20260926 14:42:00.079002 30588 leader_election.cc:288] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:42:00.079105 30593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:42:00.079299 30593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 1 LEADER]: Becoming Leader. State: Replica: 0fa183633f3540299267b4e1071c1831, State: Running, Role: LEADER
I20260926 14:42:00.079629 30593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } }
I20260926 14:42:00.080121 30588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:42:00.081207 30595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fa183633f3540299267b4e1071c1831. Latest consensus state: current_term: 1 leader_uuid: "0fa183633f3540299267b4e1071c1831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } } }
I20260926 14:42:00.081219 30594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fa183633f3540299267b4e1071c1831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } } }
I20260926 14:42:00.081413 30594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: This master's current role is: LEADER
I20260926 14:42:00.081413 30595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: This master's current role is: LEADER
I20260926 14:42:00.082075 30602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:42:00.085071 30602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:42:00.090394 30602 catalog_manager.cc:1347] Generated new cluster ID: 435dd33804b04154aef7b0c5a4d0015f
I20260926 14:42:00.090502 30602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:42:00.105041 30602 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:42:00.105989 30602 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:42:00.109930 30602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Generated new TSK 0
I20260926 14:42:00.110528 30602 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:42:00.144227 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:37775
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:42:00.273958 30614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:00.274196 30614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:00.274236 30614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:00.278173 30614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:00.278234 30614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:00.278288 30614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:42:00.283417 30614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:00.284437 30614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:00.285425 30614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:00.294584 30619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:00.294589 30620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:00.294732 30614 server_base.cc:1034] running on GCE node
W20260926 14:42:00.294623 30622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:00.295245 30614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:00.295895 30614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:00.297046 30614 hybrid_clock.cc:648] HybridClock initialized: now 1790433720297021 us; error 37 us; skew 500 ppm
Sep 26 14:42:00 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:00.299587 30614 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:42:00.300421 30614 webserver.cc:460] Webserver started at http://127.16.90.193:37121/ using document root <none> and password file <none>
I20260926 14:42:00.300920 30614 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:00.300981 30614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:00.301180 30614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:42:00.303126 30614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "7da46a3821a1409fb679a799d43815c3"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "9adc450bbf34c05a60cdd09a3e743f52"
server_key_iv: "0d3a10ffe2cdd51db0d1e7d7b196c9cf"
server_key_version: "encryptionkey@0"
I20260926 14:42:00.303637 30614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "7da46a3821a1409fb679a799d43815c3"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "9adc450bbf34c05a60cdd09a3e743f52"
server_key_iv: "0d3a10ffe2cdd51db0d1e7d7b196c9cf"
server_key_version: "encryptionkey@0"
I20260926 14:42:00.307097 30614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:42:00.309453 30629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:00.310596 30614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:42:00.310704 30614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "7da46a3821a1409fb679a799d43815c3"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "9adc450bbf34c05a60cdd09a3e743f52"
server_key_iv: "0d3a10ffe2cdd51db0d1e7d7b196c9cf"
server_key_version: "encryptionkey@0"
I20260926 14:42:00.310791 30614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:00.320869 30614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:00.323846 30614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:00.324014 30614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:00.324581 30614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:00.325341 30614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:42:00.325390 30614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:00.325441 30614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:42:00.325469 30614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:00.333770 30614 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:37041
I20260926 14:42:00.333820 30743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:37041 every 8 connection(s)
I20260926 14:42:00.334640 30614 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:42:00.340478 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 30614
I20260926 14:42:00.340626 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:42:00.340898 16747 external_mini_cluster.cc:1428] Setting key b0f66f21951eea704ae7fab0145e1578
Sep 26 14:42:00 dist-test-slave-x4sn krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:00.348562 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.336371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:59381 (local address 127.16.90.254:37775)
0926 14:42:00.336502 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:00.336506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:00.337339 (+ 833us) server_negotiation.cc:407] Connection header received
0926 14:42:00.338606 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:00.338609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:00.338658 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:00.338735 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:00.339864 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.340519 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.341688 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.342032 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.342138 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.344888 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:00.344893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:00.344896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:00.344922 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:00.346246 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:00.346885 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:00.346887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:00.346889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:00.346966 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:00.347349 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:00.347352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:00.347353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:00.347588 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:00.347722 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:00.348259 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 14:42:00.348397 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:42:00.349360 30746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.336753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:59381)
0926 14:42:00.337181 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:00.337230 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:00.338390 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:00.338879 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:00.338886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:00.339246 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:00.339747 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:00.339762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.340655 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.340658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:00.341569 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:00.341578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.342191 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.342194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:00.342378 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.342952 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:00.342971 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:00.344695 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:00.346379 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:00.346386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:00.346402 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:00.346781 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:00.347053 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:00.347057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:00.347059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:00.347237 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:00.347748 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:00.347753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:00.348013 (+ 260us) client_negotiation.cc:769] Sending connection context
0926 14:42:00.348194 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 14:42:00.348478 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
I20260926 14:42:00.350694 30744 heartbeater.cc:347] Connected to a master server at 127.16.90.254:37775
I20260926 14:42:00.350951 30744 heartbeater.cc:464] Registering TS with master...
I20260926 14:42:00.351462 30744 heartbeater.cc:511] Master 127.16.90.254:37775 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
I20260926 14:42:00.352991 30552 ts_manager.cc:194] Registered new tserver with Master: 7da46a3821a1409fb679a799d43815c3 (127.16.90.193:37041)
I20260926 14:42:00.354229 30552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:59381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:42:00.367069 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:37775
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:42:00.472926 30749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:00.473194 30749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:00.473255 30749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:00.477195 30749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:00.477252 30749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:00.477329 30749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:42:00.482508 30749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:00.483517 30749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:00.484534 30749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:00.493503 30757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:00.493492 30754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:00.493726 30749 server_base.cc:1034] running on GCE node
W20260926 14:42:00.493495 30755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:00.494199 30749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:00.494778 30749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:00.495923 30749 hybrid_clock.cc:648] HybridClock initialized: now 1790433720495904 us; error 30 us; skew 500 ppm
Sep 26 14:42:00 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:00.498361 30749 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:42:00.499184 30749 webserver.cc:460] Webserver started at http://127.16.90.194:33513/ using document root <none> and password file <none>
I20260926 14:42:00.499725 30749 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:00.499794 30749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:00.500000 30749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:42:00.501983 30749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "a9fdebfebbca468789852674d04e1b70"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "120c5965ed838a9fc86aec8402a828b3"
server_key_iv: "0cc6787998dca133662b37d79f483cf2"
server_key_version: "encryptionkey@0"
I20260926 14:42:00.502547 30749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "a9fdebfebbca468789852674d04e1b70"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "120c5965ed838a9fc86aec8402a828b3"
server_key_iv: "0cc6787998dca133662b37d79f483cf2"
server_key_version: "encryptionkey@0"
I20260926 14:42:00.506069 30749 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:42:00.508352 30764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:00.509596 30749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:42:00.509720 30749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "a9fdebfebbca468789852674d04e1b70"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "120c5965ed838a9fc86aec8402a828b3"
server_key_iv: "0cc6787998dca133662b37d79f483cf2"
server_key_version: "encryptionkey@0"
I20260926 14:42:00.509820 30749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:00.523837 30749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:00.526585 30749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:00.526759 30749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:00.527310 30749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:00.528200 30749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:42:00.528250 30749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:00.528301 30749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:42:00.528327 30749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:00.536772 30749 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:38867
I20260926 14:42:00.536793 30878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:38867 every 8 connection(s)
I20260926 14:42:00.537629 30749 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:42:00.542798 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 30749
I20260926 14:42:00.542920 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:42:00.543218 16747 external_mini_cluster.cc:1428] Setting key 3826734fc7a9a0b5e240c6ae28820299
Sep 26 14:42:00 dist-test-slave-x4sn krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:00.551076 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.539390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:58821 (local address 127.16.90.254:37775)
0926 14:42:00.539550 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:00.539555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:00.540419 (+ 864us) server_negotiation.cc:407] Connection header received
0926 14:42:00.541691 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:00.541693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:00.541750 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:00.541822 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:00.543113 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.543755 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.545117 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.545339 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.545415 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.547855 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:00.547858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:00.547860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:00.547882 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:00.549192 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:00.549651 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:00.549655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:00.549656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:00.549732 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:00.550037 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:00.550040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:00.550042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:00.550228 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:00.550313 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:00.550794 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 14:42:00.550917 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
I20260926 14:42:00.552049 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.539833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:58821)
0926 14:42:00.540269 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:00.540313 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:00.541494 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:00.541971 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:00.541977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:00.542324 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:00.542972 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:00.542993 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.543867 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.543871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:00.545004 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:00.545015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.545471 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.545474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:00.545670 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.546277 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:00.546298 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:00.547712 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:00.549296 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:00.549302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:00.549316 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:00.549561 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:00.549817 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:00.549820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:00.549822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:00.549954 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:00.550326 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:00.550332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:00.550566 (+ 234us) client_negotiation.cc:769] Sending connection context
0926 14:42:00.550736 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 14:42:00.551051 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":121,"threads_started":1}
I20260926 14:42:00.553095 30879 heartbeater.cc:347] Connected to a master server at 127.16.90.254:37775
I20260926 14:42:00.553321 30879 heartbeater.cc:464] Registering TS with master...
I20260926 14:42:00.553902 30879 heartbeater.cc:511] Master 127.16.90.254:37775 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
I20260926 14:42:00.555032 30552 ts_manager.cc:194] Registered new tserver with Master: a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867)
I20260926 14:42:00.555594 30552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:58821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:42:00.567785 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:37775
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:42:00.673898 30884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:00.674124 30884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:00.674162 30884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:00.678017 30884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:00.678068 30884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:00.678121 30884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:42:00.683207 30884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:00.684145 30884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:00.685056 30884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:00.693745 30892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:00.693789 30889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:00.693745 30890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:00.694054 30884 server_base.cc:1034] running on GCE node
I20260926 14:42:00.694407 30884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:00.694959 30884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:00.696111 30884 hybrid_clock.cc:648] HybridClock initialized: now 1790433720696092 us; error 31 us; skew 500 ppm
Sep 26 14:42:00 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:00.698527 30884 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:42:00.699316 30884 webserver.cc:460] Webserver started at http://127.16.90.195:33767/ using document root <none> and password file <none>
I20260926 14:42:00.699859 30884 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:00.699926 30884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:00.700137 30884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:42:00.702054 30884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "b36d445cb80c493abd3646e11972ce4a"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "44985776ffce2f7c031cb3d9145ce5ee"
server_key_iv: "3799a3d7885c3d329f0b59fffaf82fbb"
server_key_version: "encryptionkey@0"
I20260926 14:42:00.702574 30884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "b36d445cb80c493abd3646e11972ce4a"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "44985776ffce2f7c031cb3d9145ce5ee"
server_key_iv: "3799a3d7885c3d329f0b59fffaf82fbb"
server_key_version: "encryptionkey@0"
I20260926 14:42:00.706044 30884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:42:00.708423 30899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:00.709566 30884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:42:00.709671 30884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "b36d445cb80c493abd3646e11972ce4a"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "44985776ffce2f7c031cb3d9145ce5ee"
server_key_iv: "3799a3d7885c3d329f0b59fffaf82fbb"
server_key_version: "encryptionkey@0"
I20260926 14:42:00.709758 30884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:00.727803 30884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:00.730638 30884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:00.730806 30884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:00.731386 30884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:00.732216 30884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:42:00.732263 30884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:00.732314 30884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:42:00.732343 30884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:00.740494 30884 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:38691
I20260926 14:42:00.740522 31013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:38691 every 8 connection(s)
I20260926 14:42:00.741328 30884 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:42:00.743450 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 30884
I20260926 14:42:00.743534 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:42:00.743794 16747 external_mini_cluster.cc:1428] Setting key 6eb27d5cd5e40556293699f33e76cfc4
Sep 26 14:42:00 dist-test-slave-x4sn krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:00.753472 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.742960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:50289 (local address 127.16.90.254:37775)
0926 14:42:00.743102 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:00.743107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:00.744020 (+ 913us) server_negotiation.cc:407] Connection header received
0926 14:42:00.745316 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:00.745319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:00.745371 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:00.745449 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:00.746561 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.747083 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.748147 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.748357 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.748428 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.750574 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:00.750577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:00.750579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:00.750603 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:00.751910 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:00.752322 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:00.752324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:00.752326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:00.752393 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:00.752627 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:00.752629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:00.752630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:00.752803 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:00.752869 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:00.753269 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 14:42:00.753359 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:42:00.754237 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.743384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:50289)
0926 14:42:00.743864 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:00.743917 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:00.745138 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:00.745639 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:00.745646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:00.745975 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:00.746437 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:00.746450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.747167 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.747170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:00.748051 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:00.748059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.748474 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.748477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:00.748611 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.749079 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:00.749104 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:00.750431 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:00.751998 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:00.752002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:00.752012 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:00.752253 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:00.752450 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:00.752452 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:00.752454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:00.752568 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:00.752867 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:00.752871 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:00.753079 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 14:42:00.753241 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 14:42:00.753471 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":139,"threads_started":1}
I20260926 14:42:00.755210 31014 heartbeater.cc:347] Connected to a master server at 127.16.90.254:37775
I20260926 14:42:00.755416 31014 heartbeater.cc:464] Registering TS with master...
I20260926 14:42:00.755889 31014 heartbeater.cc:511] Master 127.16.90.254:37775 requested a full tablet report, sending...
I20260926 14:42:00.756803 30552 ts_manager.cc:194] Registered new tserver with Master: b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691)
I20260926 14:42:00.757335 30552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:50289
I20260926 14:42:00.769047 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:42:00.777999 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.770978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.16.90.254:37775)
0926 14:42:00.771136 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:00.771141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:00.771264 (+ 123us) server_negotiation.cc:407] Connection header received
0926 14:42:00.771405 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:00.771409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:00.771452 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:00.771513 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:00.772278 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.772800 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.773599 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.773823 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.773909 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.775205 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:00.775208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:00.775210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:00.775233 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:00.776493 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:00.776868 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:00.776871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:00.776872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:00.776944 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:00.777178 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:00.777181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:00.777183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:00.777398 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:00.777517 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:00.777740 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 14:42:00.777848 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:42:00.780771 30552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:42:00.783769 30552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:42:00.797475 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.791555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:37041 (local address 127.0.0.1:51034)
0926 14:42:00.791995 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:00.792047 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:00.792168 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:00.793256 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:00.793262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:00.793287 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:00.793434 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:00.793441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.794479 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.794482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:00.795870 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:00.795881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.796861 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.796865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:00.797014 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.797101 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 14:42:00.797194 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 14:42:00.797334 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":65,"thread_start_us":106,"threads_started":1}
I20260926 14:42:00.797487 31026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.791743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.16.90.193:37041)
0926 14:42:00.792158 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:00.792170 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:00.792194 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:42:00.792861 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:00.792867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:00.793080 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:00.793247 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:00.793549 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.794363 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.795984 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.796721 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.797062 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.797206 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:00.797254 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 14:42:00.797337 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":111,"threads_started":1}
I20260926 14:42:00.797544 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.792527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34738 (local address 127.16.90.194:38867)
0926 14:42:00.793218 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:00.793229 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:00.793250 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:42:00.793316 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:00.793321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:00.793495 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:00.793626 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:00.793829 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.794568 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.795872 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.796721 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.796828 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.796973 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:00.797311 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 14:42:00.797405 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":148,"threads_started":1}
I20260926 14:42:00.797858 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.791829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:38867 (local address 127.0.0.1:34738)
0926 14:42:00.792593 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:00.792606 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:00.792691 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:00.793608 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:00.793611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:00.793629 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:00.793725 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:00.793731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.794688 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.794691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:00.795755 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:00.795765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.796845 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.796849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:00.797011 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.797099 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 14:42:00.797201 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:42:00.797735 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":477,"thread_start_us":80,"threads_started":1}
I20260926 14:42:00.799053 31029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.792479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.16.90.195:38691)
0926 14:42:00.792975 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:00.792986 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:00.793008 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:42:00.793123 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:00.793128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:00.793306 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:00.793440 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:00.794737 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.795883 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.796821 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.797865 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.798084 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.798223 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:00.798832 (+ 609us) server_negotiation.cc:299] Negotiation successful
0926 14:42:00.798923 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":148,"threads_started":1}
I20260926 14:42:00.799288 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:00.792307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:38691 (local address 127.0.0.1:47002)
0926 14:42:00.792931 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:00.792943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:00.793009 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:00.793578 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:00.793581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:00.793610 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:00.793716 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:00.793723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.796014 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.796016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:00.796708 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:00.796715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:00.798279 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:00.798283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:00.798368 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:00.798391 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:42:00.798454 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:42:00.798519 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"spinlock_wait_cycles":2944,"thread_start_us":105,"threads_started":1}
I20260926 14:42:00.800595 30677 tablet_service.cc:1467] Processing CreateTablet for tablet 481d659a10634e70b7df46b091a6ac99 (DEFAULT_TABLE table=test-table [id=4efc21c4852b457699f617b1b17ee3a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:42:00.800627 30813 tablet_service.cc:1467] Processing CreateTablet for tablet 481d659a10634e70b7df46b091a6ac99 (DEFAULT_TABLE table=test-table [id=4efc21c4852b457699f617b1b17ee3a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:42:00.801550 30677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481d659a10634e70b7df46b091a6ac99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:00.801612 30813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481d659a10634e70b7df46b091a6ac99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:00.801939 30948 tablet_service.cc:1467] Processing CreateTablet for tablet 481d659a10634e70b7df46b091a6ac99 (DEFAULT_TABLE table=test-table [id=4efc21c4852b457699f617b1b17ee3a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:42:00.802842 30948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481d659a10634e70b7df46b091a6ac99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:00.807487 31031 tablet_bootstrap.cc:492] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Bootstrap starting.
I20260926 14:42:00.809002 31032 tablet_bootstrap.cc:492] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70: Bootstrap starting.
I20260926 14:42:00.809605 31031 tablet_bootstrap.cc:654] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:00.810602 31031 log.cc:824] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:00.811149 31033 tablet_bootstrap.cc:492] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a: Bootstrap starting.
I20260926 14:42:00.811189 31032 tablet_bootstrap.cc:654] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:00.812011 31032 log.cc:824] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:00.812600 31031 tablet_bootstrap.cc:492] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: No bootstrap required, opened a new log
I20260926 14:42:00.812798 31031 ts_tablet_manager.cc:1397] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:42:00.813333 31033 tablet_bootstrap.cc:654] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:00.813954 31032 tablet_bootstrap.cc:492] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70: No bootstrap required, opened a new log
I20260926 14:42:00.814014 31033 log.cc:824] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:00.814126 31032 ts_tablet_manager.cc:1397] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:42:00.815598 31033 tablet_bootstrap.cc:492] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a: No bootstrap required, opened a new log
I20260926 14:42:00.815912 31033 ts_tablet_manager.cc:1397] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:42:00.816639 31031 raft_consensus.cc:348] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } } peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } }
I20260926 14:42:00.816898 31031 raft_consensus.cc:374] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:00.816988 31031 raft_consensus.cc:729] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da46a3821a1409fb679a799d43815c3, State: Initialized, Role: FOLLOWER
I20260926 14:42:00.817430 31031 consensus_queue.cc:260] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } } peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } }
I20260926 14:42:00.817713 31032 raft_consensus.cc:348] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } } peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } }
I20260926 14:42:00.818002 31032 raft_consensus.cc:374] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:00.818090 30744 heartbeater.cc:503] Master 127.16.90.254:37775 was elected leader, sending a full tablet report...
I20260926 14:42:00.818095 31032 raft_consensus.cc:729] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9fdebfebbca468789852674d04e1b70, State: Initialized, Role: FOLLOWER
I20260926 14:42:00.818429 31031 ts_tablet_manager.cc:1428] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:42:00.818521 31032 consensus_queue.cc:260] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } } peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } }
I20260926 14:42:00.819065 30879 heartbeater.cc:503] Master 127.16.90.254:37775 was elected leader, sending a full tablet report...
I20260926 14:42:00.819003 31033 raft_consensus.cc:348] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } } peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } }
I20260926 14:42:00.819218 31033 raft_consensus.cc:374] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:00.819267 31033 raft_consensus.cc:729] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36d445cb80c493abd3646e11972ce4a, State: Initialized, Role: FOLLOWER
I20260926 14:42:00.819398 31032 ts_tablet_manager.cc:1428] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:42:00.819628 31033 consensus_queue.cc:260] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } } peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } }
I20260926 14:42:00.820488 31033 ts_tablet_manager.cc:1428] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:42:00.820796 31014 heartbeater.cc:503] Master 127.16.90.254:37775 was elected leader, sending a full tablet report...
W20260926 14:42:00.835713 30745 tablet.cc:2369] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:42:00.992280 31015 tablet.cc:2369] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:42:01.038725 30880 tablet.cc:2369] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:42:01.068037 31037 raft_consensus.cc:482] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:42:01.068224 31037 raft_consensus.cc:504] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } } peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } }
I20260926 14:42:01.069288 31037 leader_election.cc:288] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867), b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691)
I20260926 14:42:01.073333 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.069582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:38691 (local address 127.16.90.193:46821)
0926 14:42:01.069856 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:01.069871 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:01.069985 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:01.070222 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:01.070225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:01.070242 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:01.070390 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.070397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.071285 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.071288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:01.072192 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.072202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.072920 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.072924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:01.073014 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.073040 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:42:01.073108 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:42:01.073195 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260926 14:42:01.073350 31029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.069720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:46821 (local address 127.16.90.195:38691)
0926 14:42:01.069871 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:01.069875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:01.069946 (+ 71us) server_negotiation.cc:407] Connection header received
0926 14:42:01.070097 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:01.070100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:01.070141 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:01.070203 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:01.070547 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.071180 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.072321 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.072797 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.072907 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.072957 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:01.073136 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 14:42:01.073210 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 14:42:01.073503 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.069672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:48221 (local address 127.16.90.194:38867)
0926 14:42:01.069796 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:01.069801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:01.070342 (+ 541us) server_negotiation.cc:407] Connection header received
0926 14:42:01.070467 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:01.070470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:01.070523 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:01.070585 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:01.070869 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.071395 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.072550 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.072961 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.073069 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.073114 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:01.073318 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 14:42:01.073387 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:42:01.073518 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.069582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:38867 (local address 127.16.90.193:48221)
0926 14:42:01.070252 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:01.070265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:01.070372 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:01.070610 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:01.070613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:01.070632 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:01.070760 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.070768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.071502 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.071504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:01.072411 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.072422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.073075 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.073078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:01.073184 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.073210 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:42:01.073295 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:42:01.073364 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":302,"thread_start_us":166,"threads_started":1}
I20260926 14:42:01.074304 30833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481d659a10634e70b7df46b091a6ac99" candidate_uuid: "7da46a3821a1409fb679a799d43815c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9fdebfebbca468789852674d04e1b70" is_pre_election: true
I20260926 14:42:01.074304 30968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481d659a10634e70b7df46b091a6ac99" candidate_uuid: "7da46a3821a1409fb679a799d43815c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36d445cb80c493abd3646e11972ce4a" is_pre_election: true
I20260926 14:42:01.074604 30833 raft_consensus.cc:2454] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da46a3821a1409fb679a799d43815c3 in term 0.
I20260926 14:42:01.074604 30968 raft_consensus.cc:2454] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da46a3821a1409fb679a799d43815c3 in term 0.
I20260926 14:42:01.075078 30630 leader_election.cc:302] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da46a3821a1409fb679a799d43815c3, b36d445cb80c493abd3646e11972ce4a; no voters:
I20260926 14:42:01.075344 31037 raft_consensus.cc:2781] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:42:01.075428 31037 raft_consensus.cc:482] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:42:01.075497 31037 raft_consensus.cc:3037] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:01.076538 31037 raft_consensus.cc:504] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } } peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } }
I20260926 14:42:01.076850 31037 leader_election.cc:288] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 1 election: Requested vote from peers a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867), b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691)
I20260926 14:42:01.077152 30833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481d659a10634e70b7df46b091a6ac99" candidate_uuid: "7da46a3821a1409fb679a799d43815c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9fdebfebbca468789852674d04e1b70"
I20260926 14:42:01.077163 30968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481d659a10634e70b7df46b091a6ac99" candidate_uuid: "7da46a3821a1409fb679a799d43815c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36d445cb80c493abd3646e11972ce4a"
I20260926 14:42:01.077260 30968 raft_consensus.cc:3037] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:01.077265 30833 raft_consensus.cc:3037] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:01.078212 30968 raft_consensus.cc:2454] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da46a3821a1409fb679a799d43815c3 in term 1.
I20260926 14:42:01.078267 30833 raft_consensus.cc:2454] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da46a3821a1409fb679a799d43815c3 in term 1.
I20260926 14:42:01.078547 30633 leader_election.cc:302] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da46a3821a1409fb679a799d43815c3, a9fdebfebbca468789852674d04e1b70; no voters:
I20260926 14:42:01.078711 31037 raft_consensus.cc:2781] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:42:01.078903 31037 raft_consensus.cc:686] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 1 LEADER]: Becoming Leader. State: Replica: 7da46a3821a1409fb679a799d43815c3, State: Running, Role: LEADER
I20260926 14:42:01.079253 31037 consensus_queue.cc:237] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } } peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } }
I20260926 14:42:01.082386 30551 catalog_manager.cc:5495] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da46a3821a1409fb679a799d43815c3 (127.16.90.193). New cstate: current_term: 1 leader_uuid: "7da46a3821a1409fb679a799d43815c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } health_report { overall_health: UNKNOWN } } }
I20260926 14:42:01.157374 31026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.153927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.16.90.193:37041)
0926 14:42:01.154065 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:01.154070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:01.154103 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:42:01.154194 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:01.154197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:01.154254 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:01.154354 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:42:01.154602 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.155103 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.155988 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.156213 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.156304 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.156599 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:42:01.157004 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:42:01.157120 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:01.157185 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 14:42:01.157241 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:42:01.163084 30833 raft_consensus.cc:1261] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70 [term 1 FOLLOWER]: Refusing update from remote peer 7da46a3821a1409fb679a799d43815c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:42:01.163084 30968 raft_consensus.cc:1261] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a [term 1 FOLLOWER]: Refusing update from remote peer 7da46a3821a1409fb679a799d43815c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:42:01.163822 31042 consensus_queue.cc:1035] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:42:01.164001 31037 consensus_queue.cc:1035] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:42:01.172708 31046 mvcc.cc:205] Tried to move back new op lower bound from 7333616521876410368 to 7333616521543012352. Current Snapshot: MvccSnapshot[applied={T|T < 7333616521876410368}]
I20260926 14:42:01.178517 30552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:42:01.178663 30552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:42:01.180889 30552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Sending DeleteTablet for 3 replicas of tablet 481d659a10634e70b7df46b091a6ac99
I20260926 14:42:01.181854 30948 tablet_service.cc:1514] Processing DeleteTablet for tablet 481d659a10634e70b7df46b091a6ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:01 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:47002
I20260926 14:42:01.181859 30677 tablet_service.cc:1514] Processing DeleteTablet for tablet 481d659a10634e70b7df46b091a6ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:01 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:51034
I20260926 14:42:01.182276 31054 tablet_replica.cc:330] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a: stopping tablet replica
I20260926 14:42:01.182312 30813 tablet_service.cc:1514] Processing DeleteTablet for tablet 481d659a10634e70b7df46b091a6ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:01 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:34738
I20260926 14:42:01.182579 31054 raft_consensus.cc:2229] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:42:01.182722 31056 tablet_replica.cc:330] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70: stopping tablet replica
I20260926 14:42:01.182847 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 30614
I20260926 14:42:01.182865 31054 raft_consensus.cc:2258] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:42:01.183027 31056 raft_consensus.cc:2229] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:42:01.183431 31056 raft_consensus.cc:2258] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:42:01.184460 31054 ts_tablet_manager.cc:1905] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:42:01.185154 31056 ts_tablet_manager.cc:1905] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:42:01.186688 31054 ts_tablet_manager.cc:1918] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:42:01.186770 31054 log.cc:1196] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/481d659a10634e70b7df46b091a6ac99
I20260926 14:42:01.187053 31054 ts_tablet_manager.cc:1939] T 481d659a10634e70b7df46b091a6ac99 P b36d445cb80c493abd3646e11972ce4a: Deleting consensus metadata
I20260926 14:42:01.187541 31056 ts_tablet_manager.cc:1918] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:42:01.187625 31056 log.cc:1196] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/481d659a10634e70b7df46b091a6ac99
I20260926 14:42:01.187947 31056 ts_tablet_manager.cc:1939] T 481d659a10634e70b7df46b091a6ac99 P a9fdebfebbca468789852674d04e1b70: Deleting consensus metadata
I20260926 14:42:01.188069 30536 catalog_manager.cc:4841] TS b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691): tablet 481d659a10634e70b7df46b091a6ac99 (table test-table [id=4efc21c4852b457699f617b1b17ee3a6]) successfully deleted
I20260926 14:42:01.188938 30539 catalog_manager.cc:4841] TS a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867): tablet 481d659a10634e70b7df46b091a6ac99 (table test-table [id=4efc21c4852b457699f617b1b17ee3a6]) successfully deleted
W20260926 14:42:01.189831 30537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:37041 (error 108)
I20260926 14:42:01.190548 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 30749
I20260926 14:42:01.190814 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.190552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:37041 (local address 127.0.0.1:51044)
0926 14:42:01.190679 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:01.190740 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 14:42:01.195400 30537 catalog_manager.cc:4584] TS 7da46a3821a1409fb679a799d43815c3 (127.16.90.193:37041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 481d659a10634e70b7df46b091a6ac99: Network error: Client connection negotiation failed: client connection to 127.16.90.193:37041: connect: Connection refused (error 111)
I20260926 14:42:01.197238 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 30884
I20260926 14:42:01.201951 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 30520
I20260926 14:42:01.207002 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:37775
--webserver_interface=127.16.90.254
--webserver_port=44663
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:37775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:42:01.312428 31058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:01.312682 31058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:01.312734 31058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:01.316627 31058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:42:01.316681 31058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:01.316706 31058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:01.316726 31058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:42:01.316747 31058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:42:01.321834 31058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:37775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:37775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:01.322822 31058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:01.323819 31058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:42:01.337443 31058 server_base.cc:1034] running on GCE node
W20260926 14:42:01.337388 31064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:01.337381 31067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:01.337381 31065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:01.338318 31058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:01.339483 31058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:01.340667 31058 hybrid_clock.cc:648] HybridClock initialized: now 1790433721340634 us; error 42 us; skew 500 ppm
Sep 26 14:42:01 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:01.343428 31058 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:42:01.344383 31058 webserver.cc:460] Webserver started at http://127.16.90.254:44663/ using document root <none> and password file <none>
I20260926 14:42:01.344959 31058 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:01.345048 31058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:01.349076 31058 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:42:01.351096 31074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:01.352295 31058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:42:01.352407 31058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "0fa183633f3540299267b4e1071c1831"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "a32cb0c7f7b0a615d15a889e4fa2e98c"
server_key_iv: "26d150adbc38b8a3dd840b0bebf4a76d"
server_key_version: "encryptionkey@0"
I20260926 14:42:01.353036 31058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:01.377982 31058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:01.380801 31058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:01.380983 31058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:01.387933 31058 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:37775
I20260926 14:42:01.388182 31127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:37775 every 8 connection(s)
I20260926 14:42:01.389065 31058 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:42:01.392797 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 31058
I20260926 14:42:01.393260 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:37041
--local_ip_for_outbound_sockets=127.16.90.193
--tserver_master_addrs=127.16.90.254:37775
--webserver_port=37121
--webserver_interface=127.16.90.193
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:42:01.393504 31128 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:42:01.395406 31128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Bootstrap starting.
I20260926 14:42:01.411934 31128 log.cc:824] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:01.420307 31128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:42:01.420644 31128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Bootstrap complete.
I20260926 14:42:01.424511 31128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } }
I20260926 14:42:01.424898 31128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa183633f3540299267b4e1071c1831, State: Initialized, Role: FOLLOWER
I20260926 14:42:01.425364 31128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } }
I20260926 14:42:01.425506 31128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:42:01.425570 31128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:42:01.425665 31128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:42:01.427548 31128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } }
I20260926 14:42:01.427881 31128 leader_election.cc:302] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fa183633f3540299267b4e1071c1831; no voters:
I20260926 14:42:01.428175 31128 leader_election.cc:288] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:42:01.428273 31132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:42:01.428439 31132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 2 LEADER]: Becoming Leader. State: Replica: 0fa183633f3540299267b4e1071c1831, State: Running, Role: LEADER
I20260926 14:42:01.428733 31132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } }
I20260926 14:42:01.429400 31128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:42:01.429658 31132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fa183633f3540299267b4e1071c1831. Latest consensus state: current_term: 2 leader_uuid: "0fa183633f3540299267b4e1071c1831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } } }
I20260926 14:42:01.429770 31132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: This master's current role is: LEADER
I20260926 14:42:01.430115 31136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:42:01.430150 31133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0fa183633f3540299267b4e1071c1831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } } }
I20260926 14:42:01.430361 31133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: This master's current role is: LEADER
I20260926 14:42:01.434692 31136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:42:01.435300 31136 catalog_manager.cc:1259] Loaded cluster ID: 435dd33804b04154aef7b0c5a4d0015f
I20260926 14:42:01.435374 31136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:42:01.436774 31136 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 14:42:01.441537 31136 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 14:42:01.446907 31136 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:42:01.447620 31136 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Loaded TSK: 0
I20260926 14:42:01.448787 31136 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:42:01.503413 31130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:01.503682 31130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:01.503741 31130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:01.507604 31130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:01.507660 31130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:01.507756 31130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:42:01.512845 31130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:37041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=37121
--tserver_master_addrs=127.16.90.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:01.513839 31130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:01.514842 31130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:01.524925 31158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:01.524941 31155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:01.524932 31156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:01.525393 31130 server_base.cc:1034] running on GCE node
I20260926 14:42:01.525974 31130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:01.526743 31130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:01.527927 31130 hybrid_clock.cc:648] HybridClock initialized: now 1790433721527898 us; error 37 us; skew 500 ppm
Sep 26 14:42:01 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:01.531143 31130 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:42:01.532338 31130 webserver.cc:460] Webserver started at http://127.16.90.193:37121/ using document root <none> and password file <none>
I20260926 14:42:01.533013 31130 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:01.533126 31130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:01.538184 31130 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:42:01.540820 31165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:01.542335 31130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:42:01.542482 31130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "7da46a3821a1409fb679a799d43815c3"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "9adc450bbf34c05a60cdd09a3e743f52"
server_key_iv: "0d3a10ffe2cdd51db0d1e7d7b196c9cf"
server_key_version: "encryptionkey@0"
I20260926 14:42:01.543190 31130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:01.559576 31130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:01.562760 31130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:01.562935 31130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:01.563555 31130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:01.564822 31172 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:42:01.568682 31130 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:42:01.568753 31130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260926 14:42:01.568812 31130 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:42:01.571051 31130 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:42:01.571110 31130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 14:42:01.571367 31172 tablet_bootstrap.cc:492] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Bootstrap starting.
I20260926 14:42:01.580466 31130 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:37041
I20260926 14:42:01.580703 31279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:37041 every 8 connection(s)
I20260926 14:42:01.581621 31130 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:42:01.589628 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 31130
I20260926 14:42:01.590024 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:38867
--local_ip_for_outbound_sockets=127.16.90.194
--tserver_master_addrs=127.16.90.254:37775
--webserver_port=33513
--webserver_interface=127.16.90.194
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:42:01.590858 31172 log.cc:824] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:42:01 dist-test-slave-x4sn krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:01.598445 31172 tablet_bootstrap.cc:492] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:42:01.598830 31172 tablet_bootstrap.cc:492] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Bootstrap complete.
I20260926 14:42:01.600136 31172 ts_tablet_manager.cc:1397] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.008s
I20260926 14:42:01.600344 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.583814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:50153)
0926 14:42:01.584903 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:01.584956 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:01.585782 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:01.586994 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:01.587000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:01.587322 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:01.587840 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.587855 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.589792 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.589796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:01.591059 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.591070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.592047 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.592051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:01.592236 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.592782 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:01.592800 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:01.594585 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:01.597003 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:01.597008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:01.597020 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:01.597344 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:01.597732 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:01.597735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:01.597736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:01.597900 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:01.598627 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:01.598634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:01.598905 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 14:42:01.599100 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 14:42:01.599402 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":133,"threads_started":1}
I20260926 14:42:01.601272 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.583802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:50153 (local address 127.16.90.254:37775)
0926 14:42:01.584130 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:01.584149 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:01.585092 (+ 943us) server_negotiation.cc:407] Connection header received
0926 14:42:01.586195 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:01.586210 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:01.586593 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:01.586999 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:01.587991 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.589628 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.591194 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.591918 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.592062 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.594792 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:01.594799 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:01.594815 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:01.594851 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:01.596877 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:01.597459 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:01.597465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:01.597470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:01.597622 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:01.597994 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:01.597997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:01.597999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:01.598447 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:01.598581 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:01.599252 (+ 671us) server_negotiation.cc:299] Negotiation successful
0926 14:42:01.599550 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":96,"threads_started":1}
I20260926 14:42:01.602531 31172 raft_consensus.cc:348] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } } peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } }
I20260926 14:42:01.602977 31172 raft_consensus.cc:729] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da46a3821a1409fb679a799d43815c3, State: Initialized, Role: FOLLOWER
I20260926 14:42:01.603451 31172 consensus_queue.cc:260] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } } peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } }
I20260926 14:42:01.603515 31280 heartbeater.cc:347] Connected to a master server at 127.16.90.254:37775
I20260926 14:42:01.603875 31280 heartbeater.cc:464] Registering TS with master...
I20260926 14:42:01.604391 31280 heartbeater.cc:511] Master 127.16.90.254:37775 requested a full tablet report, sending...
I20260926 14:42:01.604583 31172 ts_tablet_manager.cc:1428] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260926 14:42:01.606894 31092 ts_manager.cc:194] Registered new tserver with Master: 7da46a3821a1409fb679a799d43815c3 (127.16.90.193:37041)
I20260926 14:42:01.610775 31092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:50153
I20260926 14:42:01.613025 31280 heartbeater.cc:503] Master 127.16.90.254:37775 was elected leader, sending a full tablet report...
Sep 26 14:42:01 dist-test-slave-x4sn krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:42:01.620352 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.610072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:37041 (local address 127.0.0.1:51046)
0926 14:42:01.610605 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:01.610651 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:01.610780 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:01.611315 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:01.611321 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:01.611756 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:01.611875 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.611883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.613250 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.613253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:01.614177 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.614188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.614910 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.614912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:01.614992 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.615498 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:01.615510 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:01.616989 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:01.618834 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:01.618842 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:01.618846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:01.619049 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:01.619316 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:01.619318 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:01.619319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:01.619390 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:01.619861 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:01.619866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:01.619968 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 14:42:01.620075 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 14:42:01.620228 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":119,"threads_started":1}
I20260926 14:42:01.620473 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.610135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.16.90.193:37041)
0926 14:42:01.610845 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:01.610866 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:01.610967 (+ 101us) server_negotiation.cc:407] Connection header received
0926 14:42:01.611022 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:01.611025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:01.611182 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:01.611293 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:01.611989 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.613114 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.614310 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.614803 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.614907 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.617118 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:01.617128 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:01.617140 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:01.617165 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:01.618740 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:01.619141 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:01.619147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:01.619149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:01.619238 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:01.619460 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:01.619463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:01.619465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:01.619763 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:01.619893 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:01.620164 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 14:42:01.620323 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":96,"threads_started":1}
I20260926 14:42:01.621340 31214 tablet_service.cc:1514] Processing DeleteTablet for tablet 481d659a10634e70b7df46b091a6ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:01 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:51046
I20260926 14:42:01.622135 31289 tablet_replica.cc:330] stopping tablet replica
I20260926 14:42:01.622400 31289 raft_consensus.cc:2229] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:42:01.622622 31289 raft_consensus.cc:2258] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:42:01.623513 31289 ts_tablet_manager.cc:1905] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:42:01.626971 31289 ts_tablet_manager.cc:1918] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:42:01.627050 31289 log.cc:1196] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal/wals/481d659a10634e70b7df46b091a6ac99
I20260926 14:42:01.627271 31289 ts_tablet_manager.cc:1939] T 481d659a10634e70b7df46b091a6ac99 P 7da46a3821a1409fb679a799d43815c3: Deleting consensus metadata
I20260926 14:42:01.628895 31077 catalog_manager.cc:4841] TS 7da46a3821a1409fb679a799d43815c3 (127.16.90.193:37041): tablet 481d659a10634e70b7df46b091a6ac99 (table test-table [id=4efc21c4852b457699f617b1b17ee3a6]) successfully deleted
W20260926 14:42:01.705051 31284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:01.705298 31284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:01.705353 31284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:01.709241 31284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:01.709302 31284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:01.709378 31284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:42:01.714453 31284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:38867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=33513
--tserver_master_addrs=127.16.90.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:01.715461 31284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:01.716465 31284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:01.725528 31295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:01.725487 31297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:01.725494 31294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:01.725790 31284 server_base.cc:1034] running on GCE node
I20260926 14:42:01.726274 31284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:01.726876 31284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:01.728037 31284 hybrid_clock.cc:648] HybridClock initialized: now 1790433721728019 us; error 25 us; skew 500 ppm
Sep 26 14:42:01 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:01.730643 31284 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:42:01.731459 31284 webserver.cc:460] Webserver started at http://127.16.90.194:33513/ using document root <none> and password file <none>
I20260926 14:42:01.732044 31284 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:01.732126 31284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:01.735862 31284 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 14:42:01.737829 31304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:01.738984 31284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:42:01.739094 31284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "a9fdebfebbca468789852674d04e1b70"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "120c5965ed838a9fc86aec8402a828b3"
server_key_iv: "0cc6787998dca133662b37d79f483cf2"
server_key_version: "encryptionkey@0"
I20260926 14:42:01.739621 31284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:01.770614 31284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:01.773852 31284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:01.774042 31284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:01.774629 31284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:01.775421 31284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:42:01.775476 31284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:01.775529 31284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:42:01.775554 31284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:01.783716 31284 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:38867
I20260926 14:42:01.783767 31417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:38867 every 8 connection(s)
I20260926 14:42:01.784662 31284 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:42:01.786586 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 31284
I20260926 14:42:01.786938 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:38691
--local_ip_for_outbound_sockets=127.16.90.195
--tserver_master_addrs=127.16.90.254:37775
--webserver_port=33767
--webserver_interface=127.16.90.195
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:42:01 dist-test-slave-x4sn krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:01.798317 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.786300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:50151 (local address 127.16.90.254:37775)
0926 14:42:01.786458 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:01.786463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:01.787307 (+ 844us) server_negotiation.cc:407] Connection header received
0926 14:42:01.788687 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:01.788691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:01.788752 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:01.788851 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:01.789999 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.790802 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.791988 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.792354 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.792467 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.794696 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:01.794700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:01.794703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:01.794735 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:01.796246 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:01.796810 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:01.796813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:01.796815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:01.796903 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:01.797244 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:01.797247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:01.797248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:01.797447 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:01.797535 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:01.798019 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 14:42:01.798145 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 14:42:01.799043 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.786697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:50151)
0926 14:42:01.787150 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:01.787200 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:01.788478 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:01.789031 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:01.789039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:01.789378 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:01.789849 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.789863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.790926 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.790929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:01.791877 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.791887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.792499 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.792502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:01.792642 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.793159 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:01.793175 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:01.794536 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:01.796385 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:01.796397 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:01.796415 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:01.796709 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:01.797013 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:01.797015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:01.797016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:01.797161 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:01.797589 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:01.797594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:01.797812 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 14:42:01.797971 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 14:42:01.798223 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260926 14:42:01.800104 31418 heartbeater.cc:347] Connected to a master server at 127.16.90.254:37775
I20260926 14:42:01.800315 31418 heartbeater.cc:464] Registering TS with master...
I20260926 14:42:01.800770 31418 heartbeater.cc:511] Master 127.16.90.254:37775 requested a full tablet report, sending...
I20260926 14:42:01.801725 31092 ts_manager.cc:194] Registered new tserver with Master: a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867)
I20260926 14:42:01.802481 31092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:50151
W20260926 14:42:01.893626 31421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:01.893877 31421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:01.893918 31421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:01.897774 31421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:01.897823 31421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:01.897877 31421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:42:01.902992 31421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:38691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=33767
--tserver_master_addrs=127.16.90.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:01.903954 31421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:01.904886 31421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:01.914062 31426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:01.914063 31429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:01.914064 31427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:01.914574 31421 server_base.cc:1034] running on GCE node
I20260926 14:42:01.914973 31421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:01.915575 31421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:01.916725 31421 hybrid_clock.cc:648] HybridClock initialized: now 1790433721916700 us; error 40 us; skew 500 ppm
Sep 26 14:42:01 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:01.919297 31421 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:42:01.920185 31421 webserver.cc:460] Webserver started at http://127.16.90.195:33767/ using document root <none> and password file <none>
I20260926 14:42:01.920744 31421 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:01.920840 31421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:01.924635 31421 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 14:42:01.926685 31436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:01.927891 31421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:42:01.927989 31421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "b36d445cb80c493abd3646e11972ce4a"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "44985776ffce2f7c031cb3d9145ce5ee"
server_key_iv: "3799a3d7885c3d329f0b59fffaf82fbb"
server_key_version: "encryptionkey@0"
I20260926 14:42:01.928503 31421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:01.946240 31421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:01.949501 31421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:01.949687 31421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:01.950242 31421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:01.951012 31421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:42:01.951066 31421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:01.951104 31421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:42:01.951124 31421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:01.959225 31421 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:38691
I20260926 14:42:01.959275 31549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:38691 every 8 connection(s)
I20260926 14:42:01.960210 31421 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:42:01.962999 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 31421
Sep 26 14:42:01 dist-test-slave-x4sn krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:01.971887 31554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.963640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.16.90.254:37775)
0926 14:42:01.964224 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:01.964228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:01.964241 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:42:01.964282 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:01.964285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:01.964330 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:01.964419 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:01.965228 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.965955 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.966978 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.967322 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.967414 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.968676 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:01.968679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:01.968681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:01.968702 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:01.970258 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:01.970681 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:01.970684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:01.970685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:01.970776 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:01.971057 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:01.971060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:01.971061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:01.971245 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:01.971461 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:01.971604 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 14:42:01.971740 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":79,"threads_started":1}
I20260926 14:42:01.973925 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.961822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:37767 (local address 127.16.90.254:37775)
0926 14:42:01.961971 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:01.961976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:01.962868 (+ 892us) server_negotiation.cc:407] Connection header received
0926 14:42:01.964477 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:01.964481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:01.964551 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:01.964641 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:01.965853 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.966539 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.967889 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.968083 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.968177 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.970578 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:01.970581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:01.970584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:01.970607 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:01.971979 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:01.972509 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:01.972512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:01.972514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:01.972586 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:01.972877 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:01.972879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:01.972880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:01.973036 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:01.973107 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:01.973677 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 14:42:01.973785 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:42:01.974756 31552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.962288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:37767)
0926 14:42:01.962709 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:01.962755 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:01.964278 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:01.964786 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:01.964793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:01.965115 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:01.965714 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.965732 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.966657 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.966659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:01.967788 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.967797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.968211 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.968216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:01.968378 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.968838 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:01.968856 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:01.970444 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:01.972101 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:01.972106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:01.972118 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:01.972422 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:01.972662 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:01.972665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:01.972667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:01.972799 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:01.973125 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:01.973131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:01.973438 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 14:42:01.973617 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 14:42:01.973937 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":113,"threads_started":1}
I20260926 14:42:01.975780 31550 heartbeater.cc:347] Connected to a master server at 127.16.90.254:37775
I20260926 14:42:01.975983 31550 heartbeater.cc:464] Registering TS with master...
I20260926 14:42:01.976413 31550 heartbeater.cc:511] Master 127.16.90.254:37775 requested a full tablet report, sending...
I20260926 14:42:01.977280 31092 ts_manager.cc:194] Registered new tserver with Master: b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691)
I20260926 14:42:01.977810 31092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:37767
I20260926 14:42:01.985128 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:42:01.988804 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.985674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.16.90.254:37775)
0926 14:42:01.985783 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:01.985786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:01.985961 (+ 175us) server_negotiation.cc:407] Connection header received
0926 14:42:01.986098 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:01.986100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:01.986189 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:01.986258 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:42:01.986428 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.986896 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.987639 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:01.987913 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:01.988005 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:01.988209 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:42:01.988547 (+ 338us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:42:01.988627 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:01.988673 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 14:42:01.988714 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:42:01.989337 31092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:42:01.992404 31092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:42:01.999421 31214 tablet_service.cc:1467] Processing CreateTablet for tablet 507294d0ef2e422789cab062d11e9cd4 (DEFAULT_TABLE table=test-table [id=d3bd6791679d434780d197ecb28a4c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:42:02.000119 31214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507294d0ef2e422789cab062d11e9cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:02.002799 31172 tablet_bootstrap.cc:492] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3: Bootstrap starting.
I20260926 14:42:02.003571 31172 tablet_bootstrap.cc:654] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:02.003592 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.998369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:38867 (local address 127.0.0.1:44988)
0926 14:42:01.998525 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:01.998539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:01.998628 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:01.999322 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:01.999325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:01.999343 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:01.999442 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:01.999449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.000958 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.000961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:02.002250 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:02.002260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.003176 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.003180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:02.003280 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:02.003329 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 14:42:02.003380 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 14:42:02.003458 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":3}
I20260926 14:42:02.004110 31560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.998731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:38691 (local address 127.0.0.1:51458)
0926 14:42:01.999134 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:01.999146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:01.999242 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:01.999937 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:01.999939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:01.999956 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:02.000045 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:02.000052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.001183 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.001186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:02.002576 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:02.002597 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.003762 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.003766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:02.003849 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:02.003872 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:42:02.003922 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 14:42:02.003985 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":50,"thread_start_us":90,"threads_started":1}
I20260926 14:42:02.004345 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.998552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.16.90.194:38867)
0926 14:42:01.998848 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:01.998860 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:01.998881 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:42:01.998944 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:01.998948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:01.999159 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:01.999612 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:01.999805 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.000816 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.002363 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.003036 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.003905 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:02.004053 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:02.004103 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 14:42:02.004208 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":124,"threads_started":1}
I20260926 14:42:02.004797 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:01.999169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.16.90.195:38691)
0926 14:42:01.999513 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:01.999523 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:01.999541 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:42:01.999598 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:01.999603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:01.999809 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:01.999943 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:02.000151 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.000909 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.002721 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.003597 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.004381 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:02.004521 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:02.004568 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:42:02.004654 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
I20260926 14:42:02.005383 31172 tablet_bootstrap.cc:492] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3: No bootstrap required, opened a new log
I20260926 14:42:02.005473 31172 ts_tablet_manager.cc:1397] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 14:42:02.005844 31172 raft_consensus.cc:348] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:02.005930 31172 raft_consensus.cc:374] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:02.005954 31172 raft_consensus.cc:729] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da46a3821a1409fb679a799d43815c3, State: Initialized, Role: FOLLOWER
I20260926 14:42:02.006057 31172 consensus_queue.cc:260] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:02.006309 31172 ts_tablet_manager.cc:1428] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:42:02.007385 31484 tablet_service.cc:1467] Processing CreateTablet for tablet 507294d0ef2e422789cab062d11e9cd4 (DEFAULT_TABLE table=test-table [id=d3bd6791679d434780d197ecb28a4c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:42:02.007385 31352 tablet_service.cc:1467] Processing CreateTablet for tablet 507294d0ef2e422789cab062d11e9cd4 (DEFAULT_TABLE table=test-table [id=d3bd6791679d434780d197ecb28a4c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:42:02.008509 31352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507294d0ef2e422789cab062d11e9cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:02.008517 31484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507294d0ef2e422789cab062d11e9cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:02.016494 31564 tablet_bootstrap.cc:492] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a: Bootstrap starting.
I20260926 14:42:02.016937 31563 tablet_bootstrap.cc:492] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70: Bootstrap starting.
I20260926 14:42:02.019552 31563 tablet_bootstrap.cc:654] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:02.019552 31564 tablet_bootstrap.cc:654] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:02.020483 31563 log.cc:824] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:02.020483 31564 log.cc:824] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:02.022234 31563 tablet_bootstrap.cc:492] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70: No bootstrap required, opened a new log
I20260926 14:42:02.022251 31564 tablet_bootstrap.cc:492] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a: No bootstrap required, opened a new log
I20260926 14:42:02.022410 31564 ts_tablet_manager.cc:1397] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:42:02.022406 31563 ts_tablet_manager.cc:1397] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:42:02.025161 31563 raft_consensus.cc:348] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:02.025162 31564 raft_consensus.cc:348] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:02.025365 31563 raft_consensus.cc:374] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:02.025367 31564 raft_consensus.cc:374] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:02.025401 31563 raft_consensus.cc:729] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9fdebfebbca468789852674d04e1b70, State: Initialized, Role: FOLLOWER
I20260926 14:42:02.025411 31564 raft_consensus.cc:729] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36d445cb80c493abd3646e11972ce4a, State: Initialized, Role: FOLLOWER
I20260926 14:42:02.025823 31563 consensus_queue.cc:260] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:02.025825 31564 consensus_queue.cc:260] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:02.026726 31550 heartbeater.cc:503] Master 127.16.90.254:37775 was elected leader, sending a full tablet report...
I20260926 14:42:02.026749 31418 heartbeater.cc:503] Master 127.16.90.254:37775 was elected leader, sending a full tablet report...
I20260926 14:42:02.026782 31564 ts_tablet_manager.cc:1428] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 14:42:02.026785 31563 ts_tablet_manager.cc:1428] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260926 14:42:02.035454 31419 tablet.cc:2369] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:42:02.083174 31281 tablet.cc:2369] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:42:02.139708 31286 raft_consensus.cc:482] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:42:02.139935 31286 raft_consensus.cc:504] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:02.141001 31286 leader_election.cc:288] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867), b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691)
I20260926 14:42:02.144767 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:02.141310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:56261 (local address 127.16.90.195:38691)
0926 14:42:02.141447 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:02.141451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:02.141741 (+ 290us) server_negotiation.cc:407] Connection header received
0926 14:42:02.141877 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:02.141880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:02.141918 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:02.141978 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:02.142209 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.142769 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.143857 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.144256 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.144360 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:02.144411 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:02.144585 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 14:42:02.144647 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:42:02.144797 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:02.141214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:38691 (local address 127.16.90.193:56261)
0926 14:42:02.141645 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:02.141659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:02.141783 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:02.141993 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:02.141995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:02.142022 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:02.142115 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:02.142119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.142860 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.142863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:02.143762 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:02.143773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.144363 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.144367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:02.144450 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:02.144497 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 14:42:02.144561 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:42:02.144630 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":181,"thread_start_us":47,"threads_started":1}
I20260926 14:42:02.144827 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:02.141328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:51653 (local address 127.16.90.194:38867)
0926 14:42:02.141446 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:02.141450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:02.141663 (+ 213us) server_negotiation.cc:407] Connection header received
0926 14:42:02.141836 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:02.141839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:02.141883 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:02.141945 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:02.142198 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.142805 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.143948 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.144350 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.144449 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:02.144497 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:02.144649 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 14:42:02.144712 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:42:02.144838 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:02.141214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:38867 (local address 127.16.90.193:51653)
0926 14:42:02.141567 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:02.141579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:02.141693 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:02.141961 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:02.141963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:02.141979 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:02.142102 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:02.142107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.142909 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.142912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:02.143849 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:02.143858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.144449 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.144453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:02.144535 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:02.144556 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:42:02.144627 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:42:02.144693 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":22,"spinlock_wait_cycles":36608,"thread_start_us":92,"threads_started":1}
I20260926 14:42:02.145481 31372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507294d0ef2e422789cab062d11e9cd4" candidate_uuid: "7da46a3821a1409fb679a799d43815c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9fdebfebbca468789852674d04e1b70" is_pre_election: true
I20260926 14:42:02.145711 31504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507294d0ef2e422789cab062d11e9cd4" candidate_uuid: "7da46a3821a1409fb679a799d43815c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36d445cb80c493abd3646e11972ce4a" is_pre_election: true
I20260926 14:42:02.145812 31372 raft_consensus.cc:2454] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da46a3821a1409fb679a799d43815c3 in term 0.
I20260926 14:42:02.145979 31504 raft_consensus.cc:2454] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da46a3821a1409fb679a799d43815c3 in term 0.
I20260926 14:42:02.146246 31169 leader_election.cc:302] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da46a3821a1409fb679a799d43815c3, a9fdebfebbca468789852674d04e1b70; no voters:
I20260926 14:42:02.146493 31286 raft_consensus.cc:2781] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:42:02.146577 31286 raft_consensus.cc:482] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:42:02.146623 31286 raft_consensus.cc:3037] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:02.147584 31286 raft_consensus.cc:504] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:02.147933 31286 leader_election.cc:288] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 1 election: Requested vote from peers a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867), b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691)
I20260926 14:42:02.148223 31372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507294d0ef2e422789cab062d11e9cd4" candidate_uuid: "7da46a3821a1409fb679a799d43815c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9fdebfebbca468789852674d04e1b70"
I20260926 14:42:02.148279 31504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507294d0ef2e422789cab062d11e9cd4" candidate_uuid: "7da46a3821a1409fb679a799d43815c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36d445cb80c493abd3646e11972ce4a"
I20260926 14:42:02.148322 31372 raft_consensus.cc:3037] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:02.148396 31504 raft_consensus.cc:3037] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:02.149366 31372 raft_consensus.cc:2454] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da46a3821a1409fb679a799d43815c3 in term 1.
I20260926 14:42:02.149432 31504 raft_consensus.cc:2454] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da46a3821a1409fb679a799d43815c3 in term 1.
I20260926 14:42:02.149607 31169 leader_election.cc:302] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da46a3821a1409fb679a799d43815c3, a9fdebfebbca468789852674d04e1b70; no voters:
I20260926 14:42:02.149750 31286 raft_consensus.cc:2781] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:42:02.149897 31286 raft_consensus.cc:686] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 LEADER]: Becoming Leader. State: Replica: 7da46a3821a1409fb679a799d43815c3, State: Running, Role: LEADER
I20260926 14:42:02.150136 31286 consensus_queue.cc:237] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:02.152945 31092 catalog_manager.cc:5495] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da46a3821a1409fb679a799d43815c3 (127.16.90.193). New cstate: current_term: 1 leader_uuid: "7da46a3821a1409fb679a799d43815c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } health_report { overall_health: HEALTHY } } }
I20260926 14:42:02.196576 31573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:02.193342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59678 (local address 127.16.90.193:37041)
0926 14:42:02.193661 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:02.193668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:02.193681 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:42:02.193752 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:02.193754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:02.193809 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:02.193908 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:42:02.194086 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.194613 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.195342 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:02.195551 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:02.195628 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:02.195835 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:42:02.196244 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:42:02.196326 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:02.196369 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 14:42:02.196416 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":138,"threads_started":1}
I20260926 14:42:02.200323 31504 raft_consensus.cc:1261] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a [term 1 FOLLOWER]: Refusing update from remote peer 7da46a3821a1409fb679a799d43815c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:42:02.200323 31372 raft_consensus.cc:1261] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70 [term 1 FOLLOWER]: Refusing update from remote peer 7da46a3821a1409fb679a799d43815c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:42:02.201025 31571 consensus_queue.cc:1035] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:42:02.201190 31286 consensus_queue.cc:1035] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:42:02.207396 31574 mvcc.cc:205] Tried to move back new op lower bound from 7333616526126243840 to 7333616525929431040. Current Snapshot: MvccSnapshot[applied={T|T < 7333616526126243840}]
I20260926 14:42:02.207854 31575 mvcc.cc:205] Tried to move back new op lower bound from 7333616526126243840 to 7333616525929431040. Current Snapshot: MvccSnapshot[applied={T|T < 7333616526126243840}]
I20260926 14:42:02.215400 31576 mvcc.cc:205] Tried to move back new op lower bound from 7333616526126243840 to 7333616525929431040. Current Snapshot: MvccSnapshot[applied={T|T < 7333616526126243840 or (T in {7333616526126243840})}]
I20260926 14:42:02.216447 31092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:42:02.216620 31092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:42:02.218808 31092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Sending DeleteTablet for 3 replicas of tablet 507294d0ef2e422789cab062d11e9cd4
I20260926 14:42:02.219408 31352 tablet_service.cc:1514] Processing DeleteTablet for tablet 507294d0ef2e422789cab062d11e9cd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:02 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:44988
I20260926 14:42:02.219458 31214 tablet_service.cc:1514] Processing DeleteTablet for tablet 507294d0ef2e422789cab062d11e9cd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:02 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:51046
I20260926 14:42:02.219434 31484 tablet_service.cc:1514] Processing DeleteTablet for tablet 507294d0ef2e422789cab062d11e9cd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:02 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:51458
I20260926 14:42:02.219946 31585 tablet_replica.cc:330] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70: stopping tablet replica
I20260926 14:42:02.220175 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 31130
I20260926 14:42:02.220207 31585 raft_consensus.cc:2229] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:42:02.220980 31585 raft_consensus.cc:2258] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:42:02.221314 31586 tablet_replica.cc:330] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a: stopping tablet replica
I20260926 14:42:02.221568 31586 raft_consensus.cc:2229] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:42:02.221851 31586 raft_consensus.cc:2258] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:42:02.222666 31585 ts_tablet_manager.cc:1905] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:42:02.223133 31586 ts_tablet_manager.cc:1905] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:42:02.225273 31585 ts_tablet_manager.cc:1918] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:42:02.225373 31585 log.cc:1196] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/507294d0ef2e422789cab062d11e9cd4
I20260926 14:42:02.225608 31586 ts_tablet_manager.cc:1918] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:42:02.225699 31585 ts_tablet_manager.cc:1939] T 507294d0ef2e422789cab062d11e9cd4 P a9fdebfebbca468789852674d04e1b70: Deleting consensus metadata
I20260926 14:42:02.225711 31586 log.cc:1196] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/507294d0ef2e422789cab062d11e9cd4
I20260926 14:42:02.226006 31586 ts_tablet_manager.cc:1939] T 507294d0ef2e422789cab062d11e9cd4 P b36d445cb80c493abd3646e11972ce4a: Deleting consensus metadata
I20260926 14:42:02.226977 31079 catalog_manager.cc:4841] TS a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867): tablet 507294d0ef2e422789cab062d11e9cd4 (table test-table [id=d3bd6791679d434780d197ecb28a4c53]) successfully deleted
I20260926 14:42:02.227180 31075 catalog_manager.cc:4841] TS b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691): tablet 507294d0ef2e422789cab062d11e9cd4 (table test-table [id=d3bd6791679d434780d197ecb28a4c53]) successfully deleted
W20260926 14:42:02.227360 31077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:37041 (error 108)
I20260926 14:42:02.228096 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 31284
I20260926 14:42:02.228291 31560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:02.228034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:37041 (local address 127.0.0.1:59690)
0926 14:42:02.228153 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:02.228215 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 14:42:02.233565 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 31421
W20260926 14:42:02.233500 31077 catalog_manager.cc:4584] TS 7da46a3821a1409fb679a799d43815c3 (127.16.90.193:37041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 507294d0ef2e422789cab062d11e9cd4: Network error: Client connection negotiation failed: client connection to 127.16.90.193:37041: connect: Connection refused (error 111)
I20260926 14:42:02.238168 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 31058
I20260926 14:42:02.243324 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:37775
--webserver_interface=127.16.90.254
--webserver_port=44663
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:37775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:42:02.351450 31588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:02.351760 31588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:02.351807 31588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:02.355726 31588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:42:02.355780 31588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:02.355798 31588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:02.355813 31588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:42:02.355832 31588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:42:02.361043 31588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:37775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:37775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:02.362054 31588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:02.363027 31588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:02.370980 31596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:02.371105 31593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:02.370980 31594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:02.371218 31588 server_base.cc:1034] running on GCE node
I20260926 14:42:02.371973 31588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:02.373075 31588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:02.374243 31588 hybrid_clock.cc:648] HybridClock initialized: now 1790433722374220 us; error 39 us; skew 500 ppm
Sep 26 14:42:02 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:02.376781 31588 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:42:02.377614 31588 webserver.cc:460] Webserver started at http://127.16.90.254:44663/ using document root <none> and password file <none>
I20260926 14:42:02.378118 31588 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:02.378192 31588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:02.381871 31588 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:42:02.383805 31603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:02.384944 31588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:42:02.385051 31588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "0fa183633f3540299267b4e1071c1831"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "a32cb0c7f7b0a615d15a889e4fa2e98c"
server_key_iv: "26d150adbc38b8a3dd840b0bebf4a76d"
server_key_version: "encryptionkey@0"
I20260926 14:42:02.385576 31588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:02.402941 31588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:02.405439 31588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:02.405604 31588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:02.412132 31588 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:37775
I20260926 14:42:02.412379 31655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:37775 every 8 connection(s)
I20260926 14:42:02.413120 31588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:42:02.417091 31656 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:42:02.418622 31656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Bootstrap starting.
I20260926 14:42:02.419315 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 31588
I20260926 14:42:02.419839 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:37041
--local_ip_for_outbound_sockets=127.16.90.193
--tserver_master_addrs=127.16.90.254:37775
--webserver_port=37121
--webserver_interface=127.16.90.193
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:42:02.436545 31656 log.cc:824] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:02.447986 31656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:42:02.448338 31656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831: Bootstrap complete.
I20260926 14:42:02.452101 31656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } }
I20260926 14:42:02.452492 31656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa183633f3540299267b4e1071c1831, State: Initialized, Role: FOLLOWER
I20260926 14:42:02.452940 31656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } }
I20260926 14:42:02.453053 31656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:42:02.453114 31656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:42:02.453205 31656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 2 FOLLOWER]: Advancing to term 3
I20260926 14:42:02.455051 31656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } }
I20260926 14:42:02.455363 31656 leader_election.cc:302] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fa183633f3540299267b4e1071c1831; no voters:
I20260926 14:42:02.455662 31656 leader_election.cc:288] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 14:42:02.455782 31660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 14:42:02.455972 31660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [term 3 LEADER]: Becoming Leader. State: Replica: 0fa183633f3540299267b4e1071c1831, State: Running, Role: LEADER
I20260926 14:42:02.456254 31660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } }
I20260926 14:42:02.456880 31656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:42:02.457571 31661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0fa183633f3540299267b4e1071c1831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } } }
I20260926 14:42:02.457877 31661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: This master's current role is: LEADER
I20260926 14:42:02.457795 31660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fa183633f3540299267b4e1071c1831. Latest consensus state: current_term: 3 leader_uuid: "0fa183633f3540299267b4e1071c1831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa183633f3540299267b4e1071c1831" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 37775 } } }
I20260926 14:42:02.458176 31660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa183633f3540299267b4e1071c1831 [sys.catalog]: This master's current role is: LEADER
I20260926 14:42:02.458636 31669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:42:02.463912 31669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:42:02.464541 31669 catalog_manager.cc:1259] Loaded cluster ID: 435dd33804b04154aef7b0c5a4d0015f
I20260926 14:42:02.464609 31669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 14:42:02.465572 31669 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1defbfb9ad google::LogMessage::Fail() at ??:0
@ 0x7f1defbfe88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1defbfb4de google::LogMessage::Flush() at ??:0
@ 0x7f1defbfca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1dfa53afdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1dfa53bd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1dfa538579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1dfa57c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1df1e4d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1df1e4dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1df1e4f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1df1e3e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1df34826db start_thread at ??:0
@ 0x7f1dee6f961f clone at ??:0
W20260926 14:42:02.557008 31658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:02.557375 31658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:02.557441 31658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:02.563366 31658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:02.563437 31658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:02.563531 31658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:42:02.571084 31658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:37041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=37121
--tserver_master_addrs=127.16.90.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:02.572495 31658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:02.573828 31658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:02.584450 31698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:02.584455 31700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:02.584656 31658 server_base.cc:1034] running on GCE node
W20260926 14:42:02.584666 31697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:02.585229 31658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:02.585839 31658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:02.586994 31658 hybrid_clock.cc:648] HybridClock initialized: now 1790433722586982 us; error 32 us; skew 500 ppm
Sep 26 14:42:02 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:02.589514 31658 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:42:02.590323 31658 webserver.cc:460] Webserver started at http://127.16.90.193:37121/ using document root <none> and password file <none>
I20260926 14:42:02.590834 31658 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:02.590891 31658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:02.594728 31658 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:42:02.596992 31707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:02.598305 31658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:42:02.598419 31658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "7da46a3821a1409fb679a799d43815c3"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "9adc450bbf34c05a60cdd09a3e743f52"
server_key_iv: "0d3a10ffe2cdd51db0d1e7d7b196c9cf"
server_key_version: "encryptionkey@0"
I20260926 14:42:02.598959 31658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:02.612188 31658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:02.615243 31658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:02.615425 31658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:02.616065 31658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:02.617234 31714 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:42:02.623032 31658 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:42:02.623133 31658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 14:42:02.623204 31658 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:42:02.624882 31658 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:42:02.624933 31658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260926 14:42:02.625306 31714 tablet_bootstrap.cc:492] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3: Bootstrap starting.
I20260926 14:42:02.634829 31658 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:37041
I20260926 14:42:02.635110 31821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:37041 every 8 connection(s)
I20260926 14:42:02.635852 31658 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:42:02.636236 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 31658
I20260926 14:42:02.636675 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:38867
--local_ip_for_outbound_sockets=127.16.90.194
--tserver_master_addrs=127.16.90.254:37775
--webserver_port=33513
--webserver_interface=127.16.90.194
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:42:02.646642 31714 log.cc:824] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:02.653218 31714 tablet_bootstrap.cc:492] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:42:02.653527 31714 tablet_bootstrap.cc:492] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3: Bootstrap complete.
I20260926 14:42:02.654417 31714 ts_tablet_manager.cc:1397] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.013s
I20260926 14:42:02.656711 31714 raft_consensus.cc:348] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:02.657109 31714 raft_consensus.cc:729] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da46a3821a1409fb679a799d43815c3, State: Initialized, Role: FOLLOWER
I20260926 14:42:02.657557 31714 consensus_queue.cc:260] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:02.658429 31714 ts_tablet_manager.cc:1428] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3: Time spent starting tablet: real 0.004s user 0.003s sys 0.003s
W20260926 14:42:02.708876 31825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:42:02.638726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:49045)
0926 14:42:02.639283 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:02.639338 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:02.640420 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:02.708112 (+ 67692us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":153,"threads_started":1}
W20260926 14:42:02.709437 31822 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:37775 (0 consecutive failures): Network error: Failed to ping master at 127.16.90.254:37775: Client connection negotiation failed: client connection to 127.16.90.254:37775: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:42:02.710156 31825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:02.709911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:57491)
0926 14:42:02.710032 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:02.710099 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 14:42:02.710794 31825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:02.710619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:55305)
0926 14:42:02.710707 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:02.710748 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:42:02.710947 31822 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:42:02.768404 31824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:02.768679 31824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:02.768739 31824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:02.772790 31824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:02.772853 31824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:02.772944 31824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:42:02.778246 31824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:38867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=33513
--tserver_master_addrs=127.16.90.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:02.779299 31824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:02.780364 31824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:02.789597 31835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:02.789779 31824 server_base.cc:1034] running on GCE node
W20260926 14:42:02.789624 31832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:02.789592 31833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:02.790243 31824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:02.790863 31824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:02.792014 31824 hybrid_clock.cc:648] HybridClock initialized: now 1790433722791996 us; error 31 us; skew 500 ppm
Sep 26 14:42:02 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:02.794551 31824 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:42:02.795348 31824 webserver.cc:460] Webserver started at http://127.16.90.194:33513/ using document root <none> and password file <none>
I20260926 14:42:02.795904 31824 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:02.795972 31824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:02.799554 31824 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:42:02.801496 31842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:02.802731 31824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:42:02.802821 31824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "a9fdebfebbca468789852674d04e1b70"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "120c5965ed838a9fc86aec8402a828b3"
server_key_iv: "0cc6787998dca133662b37d79f483cf2"
server_key_version: "encryptionkey@0"
I20260926 14:42:02.803337 31824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:02.819880 31824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:02.822494 31824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:02.822651 31824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:02.823173 31824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:02.823969 31824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:42:02.824019 31824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:02.824069 31824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:42:02.824098 31824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:02.832401 31824 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:38867
I20260926 14:42:02.832449 31955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:38867 every 8 connection(s)
I20260926 14:42:02.833357 31824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:42:02.834317 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 31824
I20260926 14:42:02.834741 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:38691
--local_ip_for_outbound_sockets=127.16.90.195
--tserver_master_addrs=127.16.90.254:37775
--webserver_port=33767
--webserver_interface=127.16.90.195
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:42:02.837319 31959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:02.835518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:36563)
0926 14:42:02.835994 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:02.836120 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":146,"threads_started":1}
W20260926 14:42:02.837770 31956 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:37775 (0 consecutive failures): Network error: Failed to ping master at 127.16.90.254:37775: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
I20260926 14:42:02.838354 31959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:02.838150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:43077)
0926 14:42:02.838256 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:02.838302 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 14:42:02.838905 31959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:02.838724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:56621)
0926 14:42:02.838815 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:02.838858 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 14:42:02.839061 31956 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:42:02.942530 31958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:02.942850 31958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:02.942898 31958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:02.946972 31958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:02.947046 31958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:02.947129 31958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:42:02.952432 31958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:38691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=33767
--tserver_master_addrs=127.16.90.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:02.953471 31958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:02.954461 31958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:02.963732 31964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:02.963725 31965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:02.963867 31958 server_base.cc:1034] running on GCE node
W20260926 14:42:02.964066 31967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:02.964406 31958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:02.964998 31958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:02.966147 31958 hybrid_clock.cc:648] HybridClock initialized: now 1790433722966132 us; error 29 us; skew 500 ppm
Sep 26 14:42:02 dist-test-slave-x4sn krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:02.968689 31958 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:42:02.969503 31958 webserver.cc:460] Webserver started at http://127.16.90.195:33767/ using document root <none> and password file <none>
I20260926 14:42:02.970052 31958 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:02.970120 31958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:02.973716 31958 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260926 14:42:02.975719 31974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:02.976863 31958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:42:02.976974 31958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "b36d445cb80c493abd3646e11972ce4a"
format_stamp: "Formatted at 2026-09-26 14:42:00 on dist-test-slave-x4sn"
server_key: "44985776ffce2f7c031cb3d9145ce5ee"
server_key_iv: "3799a3d7885c3d329f0b59fffaf82fbb"
server_key_version: "encryptionkey@0"
I20260926 14:42:02.977483 31958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:02.990155 31958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:02.993011 31958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:02.993176 31958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:02.993709 31958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:02.994465 31958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:42:02.994509 31958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:02.994557 31958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:42:02.994594 31958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:03.002826 31958 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:38691
I20260926 14:42:03.002852 32087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:38691 every 8 connection(s)
I20260926 14:42:03.003844 31958 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:42:03.007434 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:03.005785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:59697)
0926 14:42:03.006203 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:03.006314 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":114,"threads_started":1}
W20260926 14:42:03.007871 32088 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:37775 (0 consecutive failures): Network error: Failed to ping master at 127.16.90.254:37775: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
I20260926 14:42:03.008392 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:03.008202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:51815)
0926 14:42:03.008298 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:03.008342 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 14:42:03.008879 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:03.008723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:44851)
0926 14:42:03.008799 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:03.008835 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 14:42:03.009007 32088 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:42:03.010419 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 31958
I20260926 14:42:03.711974 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:03.711511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:43199)
0926 14:42:03.711798 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:03.711858 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":75,"threads_started":1}
I20260926 14:42:03.839953 32093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:03.839535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:50709)
0926 14:42:03.839802 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:03.839858 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":68,"threads_started":1}
I20260926 14:42:03.988130 32094 raft_consensus.cc:482] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:42:03.988340 32094 raft_consensus.cc:504] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:03.989346 32094 leader_election.cc:288] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867), b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691)
Sep 26 14:42:03 dist-test-slave-x4sn krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.195@KRBTEST.COM
Sep 26 14:42:03 dist-test-slave-x4sn krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.194@KRBTEST.COM
I20260926 14:42:04.002236 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:03.989571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:38867 (local address 127.16.90.193:56657)
0926 14:42:03.990009 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:03.990022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:03.990124 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:03.991521 (+ 1397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:03.991528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:03.992071 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:03.992588 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:03.992602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:03.993833 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:03.993835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:03.994811 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:03.994821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:03.995715 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:03.995719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:03.995893 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:03.996471 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:03.996487 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:03.998401 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:04.000309 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:04.000313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:04.000316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:04.000493 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:04.000792 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:04.000795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:04.000797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:04.000873 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:04.001410 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:04.001413 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:04.001512 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 14:42:04.001654 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 14:42:04.001993 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":304,"spinlock_wait_cycles":17664,"thread_start_us":58,"threads_started":1}
I20260926 14:42:04.002275 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:03.989729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:48825 (local address 127.16.90.195:38691)
0926 14:42:03.990143 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:03.990153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:03.990196 (+ 43us) server_negotiation.cc:407] Connection header received
0926 14:42:03.990810 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:03.990816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:03.991165 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:03.991445 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:03.992753 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:03.993705 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:03.994955 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:03.995532 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:03.995693 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:03.998286 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:03.998292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:03.998309 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:03.998336 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:03.999893 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:04.000367 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:04.000371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:04.000375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:04.000484 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:04.000790 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:04.000793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:04.000795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:04.001163 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:04.001278 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:04.001793 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 14:42:04.002058 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":117,"threads_started":1}
I20260926 14:42:04.002298 32097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:03.989735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:56657 (local address 127.16.90.194:38867)
0926 14:42:03.990064 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:03.990090 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:03.990138 (+ 48us) server_negotiation.cc:407] Connection header received
0926 14:42:03.990831 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:03.990837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:03.991165 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:03.991441 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:03.992748 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:03.993700 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:03.994939 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:03.995551 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:03.995732 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:03.998564 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:03.998568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:03.998582 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:03.998617 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:04.000206 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:04.000589 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:04.000595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:04.000600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:04.000706 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:04.000950 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:04.000954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:04.000955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:04.001310 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:04.001422 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:04.001850 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 14:42:04.002138 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":122,"threads_started":1}
I20260926 14:42:04.002262 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:03.989571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:38691 (local address 127.16.90.193:48825)
0926 14:42:03.989685 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:03.989699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:03.989807 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:03.991521 (+ 1714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:03.991527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:03.992079 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:03.992587 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:03.992602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:03.993836 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:03.993839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:03.994817 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:03.994828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:03.995646 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:03.995650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:03.995891 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:03.996520 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:03.996532 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:03.998062 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:04.000000 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:04.000005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:04.000020 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:04.000263 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:04.000570 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:04.000573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:04.000575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:04.000705 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:04.001269 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:04.001276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:04.001492 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 14:42:04.001653 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 14:42:04.002058 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":261,"spinlock_wait_cycles":62720}
W20260926 14:42:04.005393 31711 leader_election.cc:341] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867): Not found: Tablet not found: 507294d0ef2e422789cab062d11e9cd4
W20260926 14:42:04.006140 31708 leader_election.cc:341] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691): Not found: Tablet not found: 507294d0ef2e422789cab062d11e9cd4
I20260926 14:42:04.006242 31708 leader_election.cc:302] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da46a3821a1409fb679a799d43815c3; no voters: a9fdebfebbca468789852674d04e1b70, b36d445cb80c493abd3646e11972ce4a
I20260926 14:42:04.006721 32094 raft_consensus.cc:2726] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:42:04.009781 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:04.009441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:33219)
0926 14:42:04.009649 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:04.009702 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260926 14:42:04.713141 32099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:04.712694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:51329)
0926 14:42:04.712971 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:04.713029 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
I20260926 14:42:04.840924 32100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:04.840538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:35609)
0926 14:42:04.840784 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:04.840837 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260926 14:42:05.010792 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:05.010408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:60303)
0926 14:42:05.010632 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:05.010698 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260926 14:42:05.714238 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:05.713842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:50903)
0926 14:42:05.714087 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:05.714143 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260926 14:42:05.842072 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:05.841606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:35481)
0926 14:42:05.841905 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:05.841959 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
I20260926 14:42:06.011909 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:06.011486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:47367)
0926 14:42:06.011756 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:06.011817 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260926 14:42:06.093729 32105 raft_consensus.cc:482] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:42:06.093858 32105 raft_consensus.cc:504] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:06.094256 32105 leader_election.cc:288] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867), b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691)
W20260926 14:42:06.095003 31711 leader_election.cc:341] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867): Not found: Tablet not found: 507294d0ef2e422789cab062d11e9cd4
W20260926 14:42:06.095115 31708 leader_election.cc:341] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691): Not found: Tablet not found: 507294d0ef2e422789cab062d11e9cd4
I20260926 14:42:06.095181 31708 leader_election.cc:302] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da46a3821a1409fb679a799d43815c3; no voters: a9fdebfebbca468789852674d04e1b70, b36d445cb80c493abd3646e11972ce4a
I20260926 14:42:06.095300 32105 raft_consensus.cc:2726] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:42:06.715329 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:06.714934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:34011)
0926 14:42:06.715181 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:06.715233 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
I20260926 14:42:06.843116 32107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:06.842714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:47955)
0926 14:42:06.842954 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:06.843008 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20260926 14:42:07.012990 32108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:07.012594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:59211)
0926 14:42:07.012844 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:07.012899 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
I20260926 14:42:07.716709 32109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:07.716230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:37679)
0926 14:42:07.716547 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:07.716615 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":88,"threads_started":1}
I20260926 14:42:07.844223 32110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:07.843837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:39125)
0926 14:42:07.844076 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:07.844133 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260926 14:42:07.952168 32111 raft_consensus.cc:482] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:42:07.952299 32111 raft_consensus.cc:504] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:07.952664 32111 leader_election.cc:288] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867), b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691)
W20260926 14:42:07.953395 31711 leader_election.cc:341] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867): Not found: Tablet not found: 507294d0ef2e422789cab062d11e9cd4
W20260926 14:42:07.953497 31708 leader_election.cc:341] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691): Not found: Tablet not found: 507294d0ef2e422789cab062d11e9cd4
I20260926 14:42:07.953591 31708 leader_election.cc:302] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da46a3821a1409fb679a799d43815c3; no voters: a9fdebfebbca468789852674d04e1b70, b36d445cb80c493abd3646e11972ce4a
I20260926 14:42:07.953730 32111 raft_consensus.cc:2726] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:42:08.014117 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:08.013685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:60067)
0926 14:42:08.013943 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:08.014001 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260926 14:42:08.717813 32113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:08.717392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:45115)
0926 14:42:08.717664 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:08.717718 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260926 14:42:08.845208 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:08.844846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:56889)
0926 14:42:08.845070 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:08.845121 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260926 14:42:09.015367 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:09.014909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:46053)
0926 14:42:09.015208 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:09.015271 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
I20260926 14:42:09.718874 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:09.718472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:49525)
0926 14:42:09.718724 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:09.718781 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260926 14:42:09.846354 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:09.845897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:48983)
0926 14:42:09.846197 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:09.846258 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":76,"threads_started":1}
I20260926 14:42:10.016676 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:10.016190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:52443)
0926 14:42:10.016515 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:10.016578 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":83,"threads_started":1}
I20260926 14:42:10.097371 32119 raft_consensus.cc:482] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:42:10.097536 32119 raft_consensus.cc:504] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:10.098014 32119 leader_election.cc:288] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867), b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691)
W20260926 14:42:10.098789 31711 leader_election.cc:341] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867): Not found: Tablet not found: 507294d0ef2e422789cab062d11e9cd4
W20260926 14:42:10.098908 31708 leader_election.cc:341] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691): Not found: Tablet not found: 507294d0ef2e422789cab062d11e9cd4
I20260926 14:42:10.098970 31708 leader_election.cc:302] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da46a3821a1409fb679a799d43815c3; no voters: a9fdebfebbca468789852674d04e1b70, b36d445cb80c493abd3646e11972ce4a
I20260926 14:42:10.099094 32119 raft_consensus.cc:2726] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:42:10.720058 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:10.719632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:43647)
0926 14:42:10.719908 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:10.719963 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":117,"threads_started":1}
I20260926 14:42:10.847493 32121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:10.847090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:58967)
0926 14:42:10.847346 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:10.847403 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260926 14:42:11.017781 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:11.017397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:44883)
0926 14:42:11.017621 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:11.017692 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
I20260926 14:42:11.721154 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:11.720767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:33715)
0926 14:42:11.721000 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:11.721061 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
I20260926 14:42:11.848573 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:11.848159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:37803)
0926 14:42:11.848424 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:11.848478 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260926 14:42:12.018913 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:12.018445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:55973)
0926 14:42:12.018729 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:12.018797 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260926 14:42:12.722328 32126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:12.721943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:43681)
0926 14:42:12.722180 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:12.722234 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
I20260926 14:42:12.849794 32128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:12.849320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:59655)
0926 14:42:12.849641 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:12.849700 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":74,"threads_started":1}
I20260926 14:42:13.020193 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:13.019791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:48479)
0926 14:42:13.020036 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:13.020096 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260926 14:42:13.723402 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:13.722985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:34765)
0926 14:42:13.723249 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:13.723307 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":68,"threads_started":1}
I20260926 14:42:13.850841 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:13.850440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:54875)
0926 14:42:13.850691 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:13.850745 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260926 14:42:14.021294 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:14.020863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:40635)
0926 14:42:14.021132 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:14.021191 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260926 14:42:14.203460 32133 raft_consensus.cc:482] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:42:14.203588 32133 raft_consensus.cc:504] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fdebfebbca468789852674d04e1b70" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 38867 } } peers { permanent_uuid: "b36d445cb80c493abd3646e11972ce4a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 38691 } } peers { permanent_uuid: "7da46a3821a1409fb679a799d43815c3" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37041 } }
I20260926 14:42:14.203982 32133 leader_election.cc:288] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867), b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691)
W20260926 14:42:14.204697 31708 leader_election.cc:341] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b36d445cb80c493abd3646e11972ce4a (127.16.90.195:38691): Not found: Tablet not found: 507294d0ef2e422789cab062d11e9cd4
W20260926 14:42:14.204793 31711 leader_election.cc:341] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9fdebfebbca468789852674d04e1b70 (127.16.90.194:38867): Not found: Tablet not found: 507294d0ef2e422789cab062d11e9cd4
I20260926 14:42:14.204952 31711 leader_election.cc:302] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da46a3821a1409fb679a799d43815c3; no voters: a9fdebfebbca468789852674d04e1b70, b36d445cb80c493abd3646e11972ce4a
I20260926 14:42:14.205088 32133 raft_consensus.cc:2726] T 507294d0ef2e422789cab062d11e9cd4 P 7da46a3821a1409fb679a799d43815c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:42:14.724567 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:14.724170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:43625)
0926 14:42:14.724416 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:14.724471 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260926 14:42:14.851965 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:14.851534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:58727)
0926 14:42:14.851811 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:14.851869 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260926 14:42:15.022374 32136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:15.021958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:49973)
0926 14:42:15.022227 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:15.022282 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260926 14:42:15.725673 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:15.725292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:52039)
0926 14:42:15.725517 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:15.725573 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260926 14:42:15.853070 32138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:15.852679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:56069)
0926 14:42:15.852912 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:15.852971 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260926 14:42:16.023517 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:16.023078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:38069)
0926 14:42:16.023356 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:16.023418 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260926 14:42:16.726774 32140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:16.726335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:47413)
0926 14:42:16.726622 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:16.726678 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
I20260926 14:42:16.854161 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:16.853754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:52519)
0926 14:42:16.854005 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:16.854066 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260926 14:42:17.024672 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:17.024256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.195:56395)
0926 14:42:17.024510 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:17.024573 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
I20260926 14:42:17.728158 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:17.727717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.193:44227)
0926 14:42:17.727983 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:17.728040 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":68,"threads_started":1}
I20260926 14:42:17.855338 32144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:17.854942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:37775 (local address 127.16.90.194:35549)
0926 14:42:17.855184 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:17.855240 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
I20260926 14:42:18.012599 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 31658
I20260926 14:42:18.017800 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 31824
I20260926 14:42:18.022511 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 31958
2026-09-26T14:42:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19200 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:42:18 dist-test-slave-x4sn krb5kdc[32147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:42:18 dist-test-slave-x4sn krb5kdc[32147](info): set up 2 sockets
Sep 26 14:42:18 dist-test-slave-x4sn krb5kdc[32147](info): commencing operation
krb5kdc: starting...
W20260926 14:42:18.977159 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:42:19 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:42:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:42:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:42:19.091352 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:38477
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:38477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:42:19.196738 32160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:19.197017 32160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:19.197073 32160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:19.201040 32160 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 14:42:19.201093 32160 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 14:42:19.201112 32160 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 14:42:19.201128 32160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:42:19.201152 32160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:19.201185 32160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:19.201215 32160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:42:19.201237 32160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:42:19.206496 32160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:38477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:19.207743 32160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:19.208742 32160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:19.217123 32168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:19.217279 32165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:19.217197 32166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:19.217168 32160 server_base.cc:1034] running on GCE node
I20260926 14:42:19.217926 32160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:19.219056 32160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:19.220218 32160 hybrid_clock.cc:648] HybridClock initialized: now 1790433739220190 us; error 41 us; skew 500 ppm
Sep 26 14:42:19 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:19.222806 32160 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:42:19.223620 32160 webserver.cc:460] Webserver started at http://127.16.90.254:43943/ using document root <none> and password file <none>
I20260926 14:42:19.224167 32160 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:19.224232 32160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:19.224398 32160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:42:19.226358 32160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "fc844351c82d4bc8bfa051d1eb47402a"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "aefcafd4328e43885d9967ecd1245c67"
server_key_iv: "5285ffacaa00333e5c70640a8cd330cd"
server_key_version: "encryptionkey@0"
I20260926 14:42:19.226827 32160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "fc844351c82d4bc8bfa051d1eb47402a"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "aefcafd4328e43885d9967ecd1245c67"
server_key_iv: "5285ffacaa00333e5c70640a8cd330cd"
server_key_version: "encryptionkey@0"
I20260926 14:42:19.230304 32160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:42:19.232594 32175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:19.233738 32160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:42:19.233847 32160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "fc844351c82d4bc8bfa051d1eb47402a"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "aefcafd4328e43885d9967ecd1245c67"
server_key_iv: "5285ffacaa00333e5c70640a8cd330cd"
server_key_version: "encryptionkey@0"
I20260926 14:42:19.233952 32160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:19.249989 32160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:19.252717 32160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:19.252887 32160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:19.259531 32160 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:38477
I20260926 14:42:19.259827 32227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:38477 every 8 connection(s)
I20260926 14:42:19.260500 32160 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:42:19.263171 32228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:19.267125 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 32160
I20260926 14:42:19.267241 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:42:19.267505 16747 external_mini_cluster.cc:1428] Setting key 84d685fe18a469a277b34dc6fb0e764d
I20260926 14:42:19.268721 32228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Bootstrap starting.
I20260926 14:42:19.271389 32228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:19.272352 32228 log.cc:824] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:19.274226 32228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: No bootstrap required, opened a new log
Sep 26 14:42:19 dist-test-slave-x4sn krb5kdc[32147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:19.277750 32228 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } }
I20260926 14:42:19.278030 32228 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:19.278093 32228 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc844351c82d4bc8bfa051d1eb47402a, State: Initialized, Role: FOLLOWER
I20260926 14:42:19.278573 32228 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } }
I20260926 14:42:19.278716 32228 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:42:19.278782 32228 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:42:19.278873 32228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:19.279747 32228 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } }
I20260926 14:42:19.280099 32228 leader_election.cc:302] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc844351c82d4bc8bfa051d1eb47402a; no voters:
I20260926 14:42:19.280421 32228 leader_election.cc:288] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:42:19.280525 32233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:42:19.280755 32233 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 1 LEADER]: Becoming Leader. State: Replica: fc844351c82d4bc8bfa051d1eb47402a, State: Running, Role: LEADER
I20260926 14:42:19.281065 32233 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } }
I20260926 14:42:19.281726 32228 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:42:19.282451 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.268843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.16.90.254:38477)
0926 14:42:19.269247 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:19.269263 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:19.269298 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:42:19.270120 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:19.270144 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:19.270587 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:19.270946 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:19.272358 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.273512 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.274371 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.274845 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.274987 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.277429 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:19.277434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:19.277448 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:19.277474 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:19.279267 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:19.279706 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:19.279711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:19.279717 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:19.279830 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:19.280122 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:19.280124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:19.280125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:19.280635 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:19.280830 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:19.281166 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 14:42:19.281464 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":130,"threads_started":1}
I20260926 14:42:19.282970 32234 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc844351c82d4bc8bfa051d1eb47402a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } } }
I20260926 14:42:19.283104 32234 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: This master's current role is: LEADER
I20260926 14:42:19.282953 32235 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc844351c82d4bc8bfa051d1eb47402a. Latest consensus state: current_term: 1 leader_uuid: "fc844351c82d4bc8bfa051d1eb47402a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } } }
I20260926 14:42:19.283183 32235 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: This master's current role is: LEADER
I20260926 14:42:19.283492 32243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:42:19.286710 32243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:42:19.294494 32243 catalog_manager.cc:1347] Generated new cluster ID: 7281af1ddd674f93bc2bc6bfdf071722
I20260926 14:42:19.294564 32243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:42:19.312101 32243 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:42:19.313059 32243 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:42:19.324213 32243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Generated new TSK 0
I20260926 14:42:19.325110 32243 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:42:19.355741 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38477
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:42:19.463058 32254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:19.463316 32254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:19.463377 32254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:19.467316 32254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:19.467373 32254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:19.467453 32254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:42:19.472554 32254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:19.473548 32254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:19.474521 32254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:19.483623 32262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:19.483657 32260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:19.483618 32259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:19.483893 32254 server_base.cc:1034] running on GCE node
I20260926 14:42:19.484328 32254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:19.484939 32254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:19.486094 32254 hybrid_clock.cc:648] HybridClock initialized: now 1790433739486088 us; error 43 us; skew 500 ppm
Sep 26 14:42:19 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:19.488811 32254 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:42:19.489668 32254 webserver.cc:460] Webserver started at http://127.16.90.193:39373/ using document root <none> and password file <none>
I20260926 14:42:19.490206 32254 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:19.490272 32254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:19.490455 32254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:42:19.492483 32254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "1228026955334c0ca1abd5e20e62c314"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "bff2e731336969b7a9bb95d4dd72b19d"
server_key_iv: "161e9a6665e2781cab401e42e769e500"
server_key_version: "encryptionkey@0"
I20260926 14:42:19.493005 32254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "1228026955334c0ca1abd5e20e62c314"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "bff2e731336969b7a9bb95d4dd72b19d"
server_key_iv: "161e9a6665e2781cab401e42e769e500"
server_key_version: "encryptionkey@0"
I20260926 14:42:19.496564 32254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:42:19.498896 32269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:19.500111 32254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:42:19.500229 32254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "1228026955334c0ca1abd5e20e62c314"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "bff2e731336969b7a9bb95d4dd72b19d"
server_key_iv: "161e9a6665e2781cab401e42e769e500"
server_key_version: "encryptionkey@0"
I20260926 14:42:19.500322 32254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:19.524056 32254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:19.527277 32254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:19.527467 32254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:19.528083 32254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:19.528899 32254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:42:19.528947 32254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:19.528999 32254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:42:19.529028 32254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:19.537281 32254 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:46765
I20260926 14:42:19.537297 32383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:46765 every 8 connection(s)
I20260926 14:42:19.538194 32254 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:42:19.541798 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 32254
I20260926 14:42:19.541931 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:42:19.542245 16747 external_mini_cluster.cc:1428] Setting key 95d8cd1b1943439d8391bffef7589bb7
Sep 26 14:42:19 dist-test-slave-x4sn krb5kdc[32147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
I20260926 14:42:19.551527 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.539993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:46507 (local address 127.16.90.254:38477)
0926 14:42:19.540130 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:19.540134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:19.541164 (+ 1030us) server_negotiation.cc:407] Connection header received
0926 14:42:19.542072 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:19.542076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:19.542131 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:19.542231 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:19.543597 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.544173 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.545296 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.545527 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.545613 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.548220 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:19.548224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:19.548226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:19.548251 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:19.549548 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:19.550027 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:19.550030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:19.550033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:19.550129 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:19.550445 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:19.550448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:19.550450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:19.550632 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:19.550719 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:19.551238 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 14:42:19.551363 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:42:19.552443 32386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.540369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:46507)
0926 14:42:19.541009 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:19.541049 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:19.541843 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:19.542449 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:19.542456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:19.542817 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:19.543439 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:19.543459 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.544296 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.544300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:19.545172 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:19.545181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.545666 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.545669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:19.545825 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.546413 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:19.546430 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:19.548044 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:19.549663 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:19.549669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:19.549685 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:19.549929 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:19.550218 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:19.550221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:19.550223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:19.550356 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:19.550733 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:19.550740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:19.551008 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 14:42:19.551180 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 14:42:19.551505 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
I20260926 14:42:19.553778 32384 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38477
I20260926 14:42:19.554013 32384 heartbeater.cc:464] Registering TS with master...
I20260926 14:42:19.554509 32384 heartbeater.cc:511] Master 127.16.90.254:38477 requested a full tablet report, sending...
I20260926 14:42:19.556092 32192 ts_manager.cc:194] Registered new tserver with Master: 1228026955334c0ca1abd5e20e62c314 (127.16.90.193:46765)
I20260926 14:42:19.557312 32192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:46507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:42:19.568190 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38477
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:42:19.674881 32389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:19.675156 32389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:19.675218 32389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:19.679185 32389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:19.679256 32389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:19.679337 32389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:42:19.684688 32389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:19.685714 32389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:19.686693 32389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:19.695619 32394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:19.695796 32389 server_base.cc:1034] running on GCE node
W20260926 14:42:19.695605 32395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:19.695585 32397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:19.696275 32389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:19.696893 32389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:19.698055 32389 hybrid_clock.cc:648] HybridClock initialized: now 1790433739698035 us; error 30 us; skew 500 ppm
Sep 26 14:42:19 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:19.700654 32389 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:42:19.701476 32389 webserver.cc:460] Webserver started at http://127.16.90.194:46033/ using document root <none> and password file <none>
I20260926 14:42:19.701997 32389 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:19.702055 32389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:19.702250 32389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:42:19.704233 32389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "6989f5b480a74ca4a112ce1d3e96b20c"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "cbbf87f38ea95c620cb1c311d525cd2e"
server_key_iv: "a63183783e5093c040fd18888f56989d"
server_key_version: "encryptionkey@0"
I20260926 14:42:19.704738 32389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "6989f5b480a74ca4a112ce1d3e96b20c"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "cbbf87f38ea95c620cb1c311d525cd2e"
server_key_iv: "a63183783e5093c040fd18888f56989d"
server_key_version: "encryptionkey@0"
I20260926 14:42:19.708122 32389 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:42:19.710410 32404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:19.711549 32389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:42:19.711650 32389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "6989f5b480a74ca4a112ce1d3e96b20c"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "cbbf87f38ea95c620cb1c311d525cd2e"
server_key_iv: "a63183783e5093c040fd18888f56989d"
server_key_version: "encryptionkey@0"
I20260926 14:42:19.711773 32389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:19.724749 32389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:19.727735 32389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:19.727905 32389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:19.728461 32389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:19.729230 32389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:42:19.729274 32389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:19.729315 32389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:42:19.729346 32389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:19.737695 32389 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:43005
I20260926 14:42:19.737723 32517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:43005 every 8 connection(s)
I20260926 14:42:19.738520 32389 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:42:19.743875 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 32389
I20260926 14:42:19.743985 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:42:19.744256 16747 external_mini_cluster.cc:1428] Setting key e195add9a4837648269be93bff0fe704
Sep 26 14:42:19 dist-test-slave-x4sn krb5kdc[32147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:19.751952 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.740225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:54403 (local address 127.16.90.254:38477)
0926 14:42:19.740382 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:19.740387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:19.741423 (+ 1036us) server_negotiation.cc:407] Connection header received
0926 14:42:19.742575 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:19.742578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:19.742626 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:19.742699 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:19.743963 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.744561 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.745854 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.746072 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.746158 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.748656 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:19.748659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:19.748662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:19.748683 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:19.749992 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:19.750478 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:19.750481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:19.750482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:19.750579 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:19.750898 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:19.750901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:19.750903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:19.751077 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:19.751163 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:19.751640 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 14:42:19.751793 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
I20260926 14:42:19.752815 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.740607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:54403)
0926 14:42:19.741190 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:19.741244 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:19.742369 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:19.742850 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:19.742857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:19.743248 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:19.743809 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:19.743825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.744683 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.744687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:19.745731 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:19.745741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.746210 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.746214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:19.746375 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.746942 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:19.746958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:19.748498 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:19.750111 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:19.750117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:19.750131 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:19.750379 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:19.750670 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:19.750673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:19.750675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:19.750808 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:19.751178 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:19.751185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:19.751413 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 14:42:19.751577 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 14:42:19.751921 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":106,"threads_started":1}
I20260926 14:42:19.753880 32518 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38477
I20260926 14:42:19.754140 32518 heartbeater.cc:464] Registering TS with master...
I20260926 14:42:19.754631 32518 heartbeater.cc:511] Master 127.16.90.254:38477 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
I20260926 14:42:19.755723 32192 ts_manager.cc:194] Registered new tserver with Master: 6989f5b480a74ca4a112ce1d3e96b20c (127.16.90.194:43005)
I20260926 14:42:19.756414 32192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:54403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:42:19.769910 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38477
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:42:19.875823 32523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:19.876085 32523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:19.876135 32523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:19.880034 32523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:19.880081 32523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:19.880144 32523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:42:19.885526 32523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:19.886523 32523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:19.887481 32523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:19.896649 32528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:19.896631 32531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:19.896634 32529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:19.896829 32523 server_base.cc:1034] running on GCE node
I20260926 14:42:19.897280 32523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:19.897855 32523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:19.899000 32523 hybrid_clock.cc:648] HybridClock initialized: now 1790433739898981 us; error 36 us; skew 500 ppm
Sep 26 14:42:19 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:19.901583 32523 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:42:19.902444 32523 webserver.cc:460] Webserver started at http://127.16.90.195:33095/ using document root <none> and password file <none>
I20260926 14:42:19.902925 32523 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:19.902979 32523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:19.903143 32523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:42:19.905144 32523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "1953717dba5943ce79281d05ec1a85c0"
server_key_iv: "74dff83f38fb91ca62bbe46ce9e6ea28"
server_key_version: "encryptionkey@0"
I20260926 14:42:19.905642 32523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "1953717dba5943ce79281d05ec1a85c0"
server_key_iv: "74dff83f38fb91ca62bbe46ce9e6ea28"
server_key_version: "encryptionkey@0"
I20260926 14:42:19.909085 32523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:42:19.911324 32538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:19.912657 32523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:42:19.912760 32523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "1953717dba5943ce79281d05ec1a85c0"
server_key_iv: "74dff83f38fb91ca62bbe46ce9e6ea28"
server_key_version: "encryptionkey@0"
I20260926 14:42:19.912858 32523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:19.929726 32523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:19.932684 32523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:19.932859 32523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:19.933434 32523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:19.934229 32523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:42:19.934276 32523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:19.934329 32523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:42:19.934356 32523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:19.942701 32523 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:33069
I20260926 14:42:19.942742 32651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:33069 every 8 connection(s)
I20260926 14:42:19.943547 32523 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:42:19.945842 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 32523
I20260926 14:42:19.945966 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:42:19.946250 16747 external_mini_cluster.cc:1428] Setting key 33795b57907369e45302372fc630afea
Sep 26 14:42:19 dist-test-slave-x4sn krb5kdc[32147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:19.955797 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.945229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:60551 (local address 127.16.90.254:38477)
0926 14:42:19.945373 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:19.945378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:19.946546 (+ 1168us) server_negotiation.cc:407] Connection header received
0926 14:42:19.947597 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:19.947600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:19.947646 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:19.947745 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:19.949017 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.949536 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.950591 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.950798 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.950870 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.953046 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:19.953049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:19.953051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:19.953073 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:19.954185 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:19.954602 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:19.954605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:19.954606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:19.954676 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:19.954925 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:19.954927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:19.954929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:19.955089 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:19.955162 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:19.955547 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 14:42:19.955633 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:42:19.956539 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.945663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:60551)
0926 14:42:19.946414 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:19.946456 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:19.947391 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:19.947931 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:19.947940 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:19.948435 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:19.948894 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:19.948907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.949642 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.949645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:19.950495 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:19.950503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.950904 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.950906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:19.951038 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.951521 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:19.951536 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:19.952896 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:19.954284 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:19.954289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:19.954299 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:19.954526 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:19.954742 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:19.954744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:19.954745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:19.954859 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:19.955163 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:19.955167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:19.955370 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 14:42:19.955512 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 14:42:19.955794 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":141,"threads_started":1}
I20260926 14:42:19.957525 32652 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38477
I20260926 14:42:19.957731 32652 heartbeater.cc:464] Registering TS with master...
I20260926 14:42:19.958159 32652 heartbeater.cc:511] Master 127.16.90.254:38477 requested a full tablet report, sending...
I20260926 14:42:19.959147 32191 ts_manager.cc:194] Registered new tserver with Master: 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069)
I20260926 14:42:19.959865 32191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:60551
I20260926 14:42:19.959936 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:42:19.967768 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.961776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.16.90.254:38477)
0926 14:42:19.961903 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:19.961907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:19.962084 (+ 177us) server_negotiation.cc:407] Connection header received
0926 14:42:19.962161 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:19.962163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:19.962197 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:19.962255 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:19.962818 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.963288 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.964106 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.964303 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.964372 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.965454 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:19.965457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:19.965459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:19.965477 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:19.966535 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:19.966871 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:19.966873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:19.966874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:19.966935 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:19.967118 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:19.967120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:19.967121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:19.967292 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:19.967384 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:19.967518 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 14:42:19.967611 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:42:19.970257 32191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:42:19.973165 32191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:42:19.986816 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.980788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33069 (local address 127.0.0.1:39836)
0926 14:42:19.981457 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:19.981502 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:19.981624 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:19.983063 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:19.983066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:19.983081 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:19.983160 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:19.983165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.984364 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.984367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:19.985531 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:19.985538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.986293 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.986296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:19.986399 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.986434 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 14:42:19.986539 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 14:42:19.986660 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":43,"thread_start_us":65,"threads_started":1}
I20260926 14:42:19.986828 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.980924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39836 (local address 127.16.90.195:33069)
0926 14:42:19.982679 (+ 1755us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:19.982694 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:19.982714 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:42:19.982778 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:19.982783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:19.982923 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:19.983269 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:19.983375 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.984222 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.985663 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.986180 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.986272 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.986406 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:19.986595 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 14:42:19.986691 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1661,"thread_start_us":135,"threads_started":1}
I20260926 14:42:19.986878 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.981102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:46765 (local address 127.0.0.1:49802)
0926 14:42:19.982350 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:19.982361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:19.982445 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:19.982909 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:19.982912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:19.982927 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:19.983022 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:19.983029 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.984375 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.984378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:19.985331 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:19.985340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.986477 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.986480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:19.986556 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.986577 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:42:19.986637 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 14:42:19.986683 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"mutex_wait_us":25,"thread_start_us":110,"threads_started":1}
I20260926 14:42:19.987146 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.980876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:43005 (local address 127.0.0.1:51006)
0926 14:42:19.981835 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:19.981847 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:19.981924 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:19.982434 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:19.982439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:19.982458 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:19.982611 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:19.982619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.984085 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.984089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:19.985507 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:19.985518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.986507 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.986511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:19.986606 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.986629 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:42:19.986675 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 14:42:19.986740 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":54,"thread_start_us":97,"threads_started":1}
I20260926 14:42:19.987555 32667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.981564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.16.90.194:43005)
0926 14:42:19.981967 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:19.981978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:19.982013 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:42:19.982107 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:19.982111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:19.982263 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:19.982471 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:19.982740 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.983941 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.985639 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.986372 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.987160 (+ 788us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.987282 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:19.987330 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 14:42:19.987416 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":93,"threads_started":1}
I20260926 14:42:19.989688 32586 tablet_service.cc:1467] Processing CreateTablet for tablet e4d18291d308422293a16eca9caf3ae8 (DEFAULT_TABLE table=test-table [id=df00abb095184e809d1ded2c0caf45ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:42:19.989794 32452 tablet_service.cc:1467] Processing CreateTablet for tablet e4d18291d308422293a16eca9caf3ae8 (DEFAULT_TABLE table=test-table [id=df00abb095184e809d1ded2c0caf45ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:42:19.990600 32586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d18291d308422293a16eca9caf3ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:19.990620 32452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d18291d308422293a16eca9caf3ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:19.996268 32669 tablet_bootstrap.cc:492] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e: Bootstrap starting.
I20260926 14:42:19.996693 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:19.981741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.16.90.193:46765)
0926 14:42:19.982210 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:19.982221 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:19.982448 (+ 227us) server_negotiation.cc:407] Connection header received
0926 14:42:19.982550 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:19.982555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:19.982780 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:19.982926 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:19.983143 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.984239 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.985448 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:19.986211 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:19.986313 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:19.986449 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:19.986812 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 14:42:19.996539 (+ 9727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":135,"threads_started":1}
I20260926 14:42:19.997566 32670 tablet_bootstrap.cc:492] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c: Bootstrap starting.
I20260926 14:42:19.999198 32317 tablet_service.cc:1467] Processing CreateTablet for tablet e4d18291d308422293a16eca9caf3ae8 (DEFAULT_TABLE table=test-table [id=df00abb095184e809d1ded2c0caf45ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:42:19.999785 32669 tablet_bootstrap.cc:654] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:19.999804 32670 tablet_bootstrap.cc:654] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:20.000149 32317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d18291d308422293a16eca9caf3ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:20.000645 32670 log.cc:824] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:20.000645 32669 log.cc:824] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:20.002676 32670 tablet_bootstrap.cc:492] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c: No bootstrap required, opened a new log
I20260926 14:42:20.002676 32669 tablet_bootstrap.cc:492] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e: No bootstrap required, opened a new log
I20260926 14:42:20.002859 32669 ts_tablet_manager.cc:1397] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260926 14:42:20.002861 32670 ts_tablet_manager.cc:1397] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:42:20.006259 32673 tablet_bootstrap.cc:492] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Bootstrap starting.
I20260926 14:42:20.006628 32669 raft_consensus.cc:348] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:20.006630 32670 raft_consensus.cc:348] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:20.006906 32669 raft_consensus.cc:374] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:20.006906 32670 raft_consensus.cc:374] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:20.006961 32669 raft_consensus.cc:729] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1183f6d0fb5c40c4ae69912a73c5ea9e, State: Initialized, Role: FOLLOWER
I20260926 14:42:20.006968 32670 raft_consensus.cc:729] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6989f5b480a74ca4a112ce1d3e96b20c, State: Initialized, Role: FOLLOWER
I20260926 14:42:20.007414 32669 consensus_queue.cc:260] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:20.007414 32670 consensus_queue.cc:260] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:20.008298 32652 heartbeater.cc:503] Master 127.16.90.254:38477 was elected leader, sending a full tablet report...
I20260926 14:42:20.008328 32518 heartbeater.cc:503] Master 127.16.90.254:38477 was elected leader, sending a full tablet report...
I20260926 14:42:20.008430 32669 ts_tablet_manager.cc:1428] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 14:42:20.008788 32673 tablet_bootstrap.cc:654] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:20.009014 32670 ts_tablet_manager.cc:1428] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260926 14:42:20.010094 32673 log.cc:824] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:20.012264 32673 tablet_bootstrap.cc:492] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: No bootstrap required, opened a new log
I20260926 14:42:20.012446 32673 ts_tablet_manager.cc:1397] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:42:20.015538 32673 raft_consensus.cc:348] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:20.015820 32673 raft_consensus.cc:374] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:20.015880 32673 raft_consensus.cc:729] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1228026955334c0ca1abd5e20e62c314, State: Initialized, Role: FOLLOWER
I20260926 14:42:20.016314 32673 consensus_queue.cc:260] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:20.016961 32384 heartbeater.cc:503] Master 127.16.90.254:38477 was elected leader, sending a full tablet report...
I20260926 14:42:20.017256 32673 ts_tablet_manager.cc:1428] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 14:42:20.039323 32385 tablet.cc:2369] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:42:20.127066 32677 raft_consensus.cc:482] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:42:20.127261 32677 raft_consensus.cc:504] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:20.128347 32677 leader_election.cc:288] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069), 6989f5b480a74ca4a112ce1d3e96b20c (127.16.90.194:43005)
I20260926 14:42:20.132444 32667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:20.128717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:58525 (local address 127.16.90.194:43005)
0926 14:42:20.128965 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:20.128970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:20.128998 (+ 28us) server_negotiation.cc:407] Connection header received
0926 14:42:20.129111 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:20.129114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:20.129173 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:20.129247 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:20.129548 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:20.130169 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:20.131473 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:20.131926 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:20.132038 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:20.132093 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:20.132255 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 14:42:20.132323 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260926 14:42:20.132462 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:20.128638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:43005 (local address 127.16.90.193:58525)
0926 14:42:20.128894 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:20.128907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:20.129006 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:20.129274 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:20.129277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:20.129294 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:20.129442 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:20.129449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:20.130272 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:20.130276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:20.131353 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:20.131363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:20.132033 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:20.132037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:20.132148 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:20.132168 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:42:20.132237 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:42:20.132310 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":63,"thread_start_us":93,"threads_started":1}
I20260926 14:42:20.132551 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:20.128694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:55435 (local address 127.16.90.195:33069)
0926 14:42:20.128965 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:20.128970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:20.129205 (+ 235us) server_negotiation.cc:407] Connection header received
0926 14:42:20.129302 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:20.129305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:20.129351 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:20.129435 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:20.129700 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:20.130192 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:20.131499 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:20.131995 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:20.132103 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:20.132154 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:20.132364 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 14:42:20.132433 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260926 14:42:20.132753 32679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:20.128638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33069 (local address 127.16.90.193:55435)
0926 14:42:20.129102 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:20.129115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:20.129219 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:20.129451 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:20.129453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:20.129467 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:20.129592 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:20.129599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:20.130325 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:20.130329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:20.131378 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:20.131387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:20.132107 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:20.132111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:20.132251 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:20.132269 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:42:20.132334 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:42:20.132402 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":74,"thread_start_us":87,"threads_started":1}
I20260926 14:42:20.133160 32472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d18291d308422293a16eca9caf3ae8" candidate_uuid: "1228026955334c0ca1abd5e20e62c314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" is_pre_election: true
I20260926 14:42:20.133283 32606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d18291d308422293a16eca9caf3ae8" candidate_uuid: "1228026955334c0ca1abd5e20e62c314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" is_pre_election: true
I20260926 14:42:20.133468 32472 raft_consensus.cc:2454] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1228026955334c0ca1abd5e20e62c314 in term 0.
I20260926 14:42:20.133541 32606 raft_consensus.cc:2454] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1228026955334c0ca1abd5e20e62c314 in term 0.
I20260926 14:42:20.133937 32270 leader_election.cc:302] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1228026955334c0ca1abd5e20e62c314, 6989f5b480a74ca4a112ce1d3e96b20c; no voters:
I20260926 14:42:20.134205 32677 raft_consensus.cc:2781] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:42:20.134287 32677 raft_consensus.cc:482] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:42:20.134317 32677 raft_consensus.cc:3037] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:20.135286 32677 raft_consensus.cc:504] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:20.135598 32677 leader_election.cc:288] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 1 election: Requested vote from peers 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069), 6989f5b480a74ca4a112ce1d3e96b20c (127.16.90.194:43005)
I20260926 14:42:20.135943 32606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d18291d308422293a16eca9caf3ae8" candidate_uuid: "1228026955334c0ca1abd5e20e62c314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e"
I20260926 14:42:20.135967 32472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d18291d308422293a16eca9caf3ae8" candidate_uuid: "1228026955334c0ca1abd5e20e62c314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6989f5b480a74ca4a112ce1d3e96b20c"
I20260926 14:42:20.136052 32606 raft_consensus.cc:3037] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:20.136112 32472 raft_consensus.cc:3037] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:20.137045 32606 raft_consensus.cc:2454] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1228026955334c0ca1abd5e20e62c314 in term 1.
I20260926 14:42:20.137130 32472 raft_consensus.cc:2454] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1228026955334c0ca1abd5e20e62c314 in term 1.
I20260926 14:42:20.137342 32271 leader_election.cc:302] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1183f6d0fb5c40c4ae69912a73c5ea9e, 1228026955334c0ca1abd5e20e62c314; no voters:
I20260926 14:42:20.137491 32677 raft_consensus.cc:2781] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:42:20.137708 32677 raft_consensus.cc:686] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 1 LEADER]: Becoming Leader. State: Replica: 1228026955334c0ca1abd5e20e62c314, State: Running, Role: LEADER
I20260926 14:42:20.138042 32677 consensus_queue.cc:237] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:20.141077 32191 catalog_manager.cc:5495] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1228026955334c0ca1abd5e20e62c314 (127.16.90.193). New cstate: current_term: 1 leader_uuid: "1228026955334c0ca1abd5e20e62c314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } health_report { overall_health: HEALTHY } } }
I20260926 14:42:20.175729 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:20.172488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.16.90.193:46765)
0926 14:42:20.172646 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:20.172649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:20.172663 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:42:20.172738 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:20.172740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:20.172793 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:20.172880 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:42:20.173145 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:20.173638 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:20.174456 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:20.174649 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:20.174713 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:20.174947 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:42:20.175363 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:42:20.175450 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:20.175520 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 14:42:20.175569 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 14:42:20.181492 32472 raft_consensus.cc:1261] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c [term 1 FOLLOWER]: Refusing update from remote peer 1228026955334c0ca1abd5e20e62c314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:42:20.181489 32606 raft_consensus.cc:1261] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 1 FOLLOWER]: Refusing update from remote peer 1228026955334c0ca1abd5e20e62c314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:42:20.182202 32680 consensus_queue.cc:1035] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:42:20.182375 32677 consensus_queue.cc:1035] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:42:20.192481 32685 mvcc.cc:205] Tried to move back new op lower bound from 7333616599776358400 to 7333616599608012800. Current Snapshot: MvccSnapshot[applied={T|T < 7333616599776358400}]
I20260926 14:42:20.200758 32191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:42:20.200942 32191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:42:20.203475 32191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Sending DeleteTablet for 3 replicas of tablet e4d18291d308422293a16eca9caf3ae8
I20260926 14:42:20.204372 32452 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d18291d308422293a16eca9caf3ae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:20 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:51006
I20260926 14:42:20.204378 32586 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d18291d308422293a16eca9caf3ae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:20 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:39836
I20260926 14:42:20.204514 32317 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d18291d308422293a16eca9caf3ae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:20 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:49802
I20260926 14:42:20.204924 32693 tablet_replica.cc:330] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c: stopping tablet replica
I20260926 14:42:20.205094 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 32254
I20260926 14:42:20.205188 32693 raft_consensus.cc:2229] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:42:20.205476 32693 raft_consensus.cc:2258] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:42:20.205830 32694 tablet_replica.cc:330] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e: stopping tablet replica
I20260926 14:42:20.206055 32694 raft_consensus.cc:2229] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:42:20.206266 32694 raft_consensus.cc:2258] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:42:20.206949 32693 ts_tablet_manager.cc:1905] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:42:20.207346 32694 ts_tablet_manager.cc:1905] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:42:20.209636 32694 ts_tablet_manager.cc:1918] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:42:20.209640 32693 ts_tablet_manager.cc:1918] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:42:20.209724 32694 log.cc:1196] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/e4d18291d308422293a16eca9caf3ae8
I20260926 14:42:20.209731 32693 log.cc:1196] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/e4d18291d308422293a16eca9caf3ae8
I20260926 14:42:20.210028 32693 ts_tablet_manager.cc:1939] T e4d18291d308422293a16eca9caf3ae8 P 6989f5b480a74ca4a112ce1d3e96b20c: Deleting consensus metadata
I20260926 14:42:20.210178 32694 ts_tablet_manager.cc:1939] T e4d18291d308422293a16eca9caf3ae8 P 1183f6d0fb5c40c4ae69912a73c5ea9e: Deleting consensus metadata
I20260926 14:42:20.211230 32177 catalog_manager.cc:4841] TS 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069): tablet e4d18291d308422293a16eca9caf3ae8 (table test-table [id=df00abb095184e809d1ded2c0caf45ab]) successfully deleted
I20260926 14:42:20.211246 32176 catalog_manager.cc:4841] TS 6989f5b480a74ca4a112ce1d3e96b20c (127.16.90.194:43005): tablet e4d18291d308422293a16eca9caf3ae8 (table test-table [id=df00abb095184e809d1ded2c0caf45ab]) successfully deleted
W20260926 14:42:20.213529 32178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:46765 (error 108)
I20260926 14:42:20.213789 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 32389
I20260926 14:42:20.214831 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:20.214511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:46765 (local address 127.0.0.1:49812)
0926 14:42:20.214681 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:20.214750 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260926 14:42:20.214979 32178 catalog_manager.cc:4584] TS 1228026955334c0ca1abd5e20e62c314 (127.16.90.193:46765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4d18291d308422293a16eca9caf3ae8: Network error: Client connection negotiation failed: client connection to 127.16.90.193:46765: connect: Connection refused (error 111)
I20260926 14:42:20.219802 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 32523
I20260926 14:42:20.224210 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 32160
I20260926 14:42:40.229542 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:38477
--webserver_interface=127.16.90.254
--webserver_port=43943
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:38477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:42:40.337416 32697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:40.337663 32697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:40.337718 32697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:40.341678 32697 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 14:42:40.341746 32697 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 14:42:40.341768 32697 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 14:42:40.341784 32697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:42:40.341807 32697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:40.341838 32697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:40.341867 32697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:42:40.341893 32697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:42:40.347321 32697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:38477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=43943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:40.348371 32697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:40.349408 32697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:40.362119 32706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:40.362133 32703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:40.362200 32697 server_base.cc:1034] running on GCE node
W20260926 14:42:40.362134 32704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:40.362924 32697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:40.364084 32697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:40.365242 32697 hybrid_clock.cc:648] HybridClock initialized: now 1790433760365212 us; error 45 us; skew 500 ppm
Sep 26 14:42:40 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433760, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:40.373409 32697 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:42:40.374366 32697 webserver.cc:460] Webserver started at http://127.16.90.254:43943/ using document root <none> and password file <none>
I20260926 14:42:40.374908 32697 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:40.375001 32697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:40.378974 32697 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:42:40.381165 32713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:40.382387 32697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:42:40.382503 32697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "fc844351c82d4bc8bfa051d1eb47402a"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "aefcafd4328e43885d9967ecd1245c67"
server_key_iv: "5285ffacaa00333e5c70640a8cd330cd"
server_key_version: "encryptionkey@0"
I20260926 14:42:40.383104 32697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:40.394944 32697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:40.397612 32697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:40.397794 32697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:40.404608 32697 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:38477
I20260926 14:42:40.404865 32765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:38477 every 8 connection(s)
I20260926 14:42:40.405654 32697 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:42:40.409853 32766 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:42:40.411401 32766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Bootstrap starting.
I20260926 14:42:40.415814 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 32697
I20260926 14:42:40.416265 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:46765
--local_ip_for_outbound_sockets=127.16.90.193
--tserver_master_addrs=127.16.90.254:38477
--webserver_port=39373
--webserver_interface=127.16.90.193
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:42:40.428766 32766 log.cc:824] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:40.437474 32766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:42:40.437803 32766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Bootstrap complete.
I20260926 14:42:40.441721 32766 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } }
I20260926 14:42:40.442116 32766 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc844351c82d4bc8bfa051d1eb47402a, State: Initialized, Role: FOLLOWER
I20260926 14:42:40.442561 32766 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } }
I20260926 14:42:40.442705 32766 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:42:40.442773 32766 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:42:40.442857 32766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:42:40.444763 32766 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } }
I20260926 14:42:40.445077 32766 leader_election.cc:302] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc844351c82d4bc8bfa051d1eb47402a; no voters:
I20260926 14:42:40.445364 32766 leader_election.cc:288] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:42:40.445464 302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:42:40.445673 302 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 2 LEADER]: Becoming Leader. State: Replica: fc844351c82d4bc8bfa051d1eb47402a, State: Running, Role: LEADER
I20260926 14:42:40.445961 302 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } }
I20260926 14:42:40.446621 32766 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:42:40.446926 302 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc844351c82d4bc8bfa051d1eb47402a. Latest consensus state: current_term: 2 leader_uuid: "fc844351c82d4bc8bfa051d1eb47402a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } } }
I20260926 14:42:40.447036 302 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: This master's current role is: LEADER
I20260926 14:42:40.447374 306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:42:40.447409 303 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fc844351c82d4bc8bfa051d1eb47402a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } } }
I20260926 14:42:40.447613 303 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: This master's current role is: LEADER
I20260926 14:42:40.452178 306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:42:40.452809 306 catalog_manager.cc:1259] Loaded cluster ID: 7281af1ddd674f93bc2bc6bfdf071722
I20260926 14:42:40.452874 306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:42:40.455292 306 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:42:40.456030 306 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Loaded TSK: 0
I20260926 14:42:40.470804 306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Generated new TSK 1
I20260926 14:42:40.471082 306 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:42:40.528415 300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:40.528661 300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:40.528718 300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:40.532728 300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:40.532789 300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:40.532866 300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:42:40.538057 300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:46765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=39373
--tserver_master_addrs=127.16.90.254:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:40.539028 300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:40.540035 300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:40.550242 327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:40.550370 329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:40.550235 326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:40.551021 300 server_base.cc:1034] running on GCE node
I20260926 14:42:40.551360 300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:40.552009 300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:40.553165 300 hybrid_clock.cc:648] HybridClock initialized: now 1790433760553139 us; error 44 us; skew 500 ppm
Sep 26 14:42:40 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433760, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:40.555794 300 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:42:40.556627 300 webserver.cc:460] Webserver started at http://127.16.90.193:39373/ using document root <none> and password file <none>
I20260926 14:42:40.557142 300 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:40.557224 300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:40.561076 300 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:42:40.563297 336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:40.564695 300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:42:40.564813 300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "1228026955334c0ca1abd5e20e62c314"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "bff2e731336969b7a9bb95d4dd72b19d"
server_key_iv: "161e9a6665e2781cab401e42e769e500"
server_key_version: "encryptionkey@0"
I20260926 14:42:40.565356 300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:40.580543 300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:40.583649 300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:40.583866 300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:40.584465 300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:40.585589 343 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:42:40.589612 300 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:42:40.589670 300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 14:42:40.589723 300 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:42:40.591372 300 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:42:40.591432 300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 14:42:40.591732 343 tablet_bootstrap.cc:492] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Bootstrap starting.
I20260926 14:42:40.602730 300 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:46765
I20260926 14:42:40.602823 451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:46765 every 8 connection(s)
I20260926 14:42:40.604022 300 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:42:40.612363 343 log.cc:824] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:40.612890 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 300
I20260926 14:42:40.613385 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:43005
--local_ip_for_outbound_sockets=127.16.90.194
--tserver_master_addrs=127.16.90.254:38477
--webserver_port=46033
--webserver_interface=127.16.90.194
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:42:40 dist-test-slave-x4sn krb5kdc[32147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433760, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:40.620133 343 tablet_bootstrap.cc:492] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:42:40.620532 343 tablet_bootstrap.cc:492] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Bootstrap complete.
I20260926 14:42:40.621645 343 ts_tablet_manager.cc:1397] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.004s
I20260926 14:42:40.622733 454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:40.605902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:46683)
0926 14:42:40.606556 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:40.606619 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:40.607398 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:40.608761 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:40.608768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:40.609089 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:40.609527 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:40.609540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.611507 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.611509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:40.612368 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:40.612377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.613245 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.613249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:40.613437 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:40.615179 (+ 1742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:40.615199 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:40.616879 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:40.619332 (+ 2453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:40.619338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:40.619350 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:40.619706 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:40.620110 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:40.620113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:40.620115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:40.620298 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:40.620990 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:40.620996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:40.621260 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 14:42:40.621432 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 14:42:40.621821 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":163,"threads_started":1}
I20260926 14:42:40.622816 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:40.606030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:46683 (local address 127.16.90.254:38477)
0926 14:42:40.606414 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:40.606438 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:40.606738 (+ 300us) server_negotiation.cc:407] Connection header received
0926 14:42:40.607850 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:40.607865 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:40.608291 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:40.608720 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:40.609700 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.611369 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.612486 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.613112 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.613262 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:40.617091 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:40.617097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:40.617113 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:40.617145 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:40.619205 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:40.619816 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:40.619823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:40.619829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:40.619989 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:40.620397 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:40.620400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:40.620401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:40.620867 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:40.621020 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:40.621603 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 14:42:40.621935 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":115,"threads_started":1}
I20260926 14:42:40.624418 343 raft_consensus.cc:348] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:40.624814 343 raft_consensus.cc:729] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1228026955334c0ca1abd5e20e62c314, State: Initialized, Role: FOLLOWER
I20260926 14:42:40.625264 343 consensus_queue.cc:260] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:40.625703 452 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38477
I20260926 14:42:40.625914 452 heartbeater.cc:464] Registering TS with master...
I20260926 14:42:40.626220 343 ts_tablet_manager.cc:1428] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 14:42:40.626403 452 heartbeater.cc:511] Master 127.16.90.254:38477 requested a full tablet report, sending...
I20260926 14:42:40.628773 32729 ts_manager.cc:194] Registered new tserver with Master: 1228026955334c0ca1abd5e20e62c314 (127.16.90.193:46765)
I20260926 14:42:40.632951 32729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:46683
I20260926 14:42:40.635337 452 heartbeater.cc:503] Master 127.16.90.254:38477 was elected leader, sending a full tablet report...
Sep 26 14:42:40 dist-test-slave-x4sn krb5kdc[32147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433760, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:42:40.641780 460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:40.631956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.16.90.193:46765)
0926 14:42:40.632610 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:40.632620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:40.632636 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:42:40.632684 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:40.632687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:40.632826 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:40.632936 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:40.633563 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.634526 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.635761 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.636152 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.636247 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:40.638603 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:40.638611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:40.638620 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:40.638641 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:40.640100 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:40.640578 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:40.640583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:40.640585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:40.640654 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:40.640896 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:40.640898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:40.640899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:40.641116 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:40.641219 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:40.641527 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 14:42:40.641651 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":103,"threads_started":1}
I20260926 14:42:40.641824 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:40.631905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:46765 (local address 127.0.0.1:33954)
0926 14:42:40.632256 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:40.632297 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:40.632411 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:40.632961 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:40.632967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:40.633348 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:40.633447 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:40.633454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.634649 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.634653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:40.635623 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:40.635635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.636262 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.636265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:40.636377 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:40.636976 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:40.636987 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:40.638477 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:40.640196 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:40.640205 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:40.640210 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:40.640486 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:40.640726 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:40.640728 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:40.640730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:40.640821 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:40.641210 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:40.641216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:40.641332 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 14:42:40.641472 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 14:42:40.641649 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
I20260926 14:42:40.642836 385 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d18291d308422293a16eca9caf3ae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:20 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:33954
I20260926 14:42:40.643857 461 tablet_replica.cc:330] stopping tablet replica
I20260926 14:42:40.644111 461 raft_consensus.cc:2229] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:42:40.644320 461 raft_consensus.cc:2258] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:42:40.645193 461 ts_tablet_manager.cc:1905] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:42:40.648625 461 ts_tablet_manager.cc:1918] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:42:40.648708 461 log.cc:1196] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal/wals/e4d18291d308422293a16eca9caf3ae8
I20260926 14:42:40.648941 461 ts_tablet_manager.cc:1939] T e4d18291d308422293a16eca9caf3ae8 P 1228026955334c0ca1abd5e20e62c314: Deleting consensus metadata
I20260926 14:42:40.650555 32716 catalog_manager.cc:4841] TS 1228026955334c0ca1abd5e20e62c314 (127.16.90.193:46765): tablet e4d18291d308422293a16eca9caf3ae8 (table test-table [id=df00abb095184e809d1ded2c0caf45ab]) successfully deleted
W20260926 14:42:40.729358 457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:40.729653 457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:40.729700 457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:40.733598 457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:40.733650 457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:40.733705 457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:42:40.738952 457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:43005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=46033
--tserver_master_addrs=127.16.90.254:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:40.739995 457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:40.740974 457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:42:40.750186 469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:40.750160 466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:40.750180 467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:40.750425 457 server_base.cc:1034] running on GCE node
I20260926 14:42:40.750795 457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:40.751338 457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:40.752483 457 hybrid_clock.cc:648] HybridClock initialized: now 1790433760752466 us; error 29 us; skew 500 ppm
Sep 26 14:42:40 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433760, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:40.754897 457 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:42:40.755702 457 webserver.cc:460] Webserver started at http://127.16.90.194:46033/ using document root <none> and password file <none>
I20260926 14:42:40.756201 457 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:40.756268 457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:40.759750 457 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 14:42:40.761729 476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:40.762859 457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:42:40.762959 457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "6989f5b480a74ca4a112ce1d3e96b20c"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "cbbf87f38ea95c620cb1c311d525cd2e"
server_key_iv: "a63183783e5093c040fd18888f56989d"
server_key_version: "encryptionkey@0"
I20260926 14:42:40.763460 457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:40.792689 457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:40.795347 457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:40.795521 457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:40.796131 457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:40.796931 457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:42:40.796977 457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:40.797029 457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:42:40.797055 457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:40.805546 457 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:43005
I20260926 14:42:40.805572 590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:43005 every 8 connection(s)
I20260926 14:42:40.806494 457 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:42:40.810902 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 457
I20260926 14:42:40.811288 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:33069
--local_ip_for_outbound_sockets=127.16.90.195
--tserver_master_addrs=127.16.90.254:38477
--webserver_port=33095
--webserver_interface=127.16.90.195
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:42:40 dist-test-slave-x4sn krb5kdc[32147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433760, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:40.820247 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:40.808243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:33067 (local address 127.16.90.254:38477)
0926 14:42:40.808414 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:40.808418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:40.809217 (+ 799us) server_negotiation.cc:407] Connection header received
0926 14:42:40.810499 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:40.810503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:40.810566 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:40.810658 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:40.811951 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.812488 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.813608 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.813839 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.813931 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:40.816678 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:40.816681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:40.816683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:40.816710 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:40.818143 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:40.818727 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:40.818730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:40.818731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:40.818810 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:40.819157 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:40.819159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:40.819161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:40.819315 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:40.819385 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:40.820007 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 14:42:40.820122 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 14:42:40.821311 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:40.808599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:33067)
0926 14:42:40.809042 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:40.809104 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:40.810290 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:40.810846 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:40.810853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:40.811244 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:40.811807 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:40.811823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.812609 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.812612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:40.813481 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:40.813490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.813978 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.813982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:40.814156 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:40.814669 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:40.814685 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:40.816498 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:40.818259 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:40.818265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:40.818280 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:40.818614 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:40.818898 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:40.818901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:40.818903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:40.819075 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:40.819407 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:40.819413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:40.819755 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 14:42:40.819972 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 14:42:40.820310 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":104,"threads_started":1}
I20260926 14:42:40.822396 591 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38477
I20260926 14:42:40.822610 591 heartbeater.cc:464] Registering TS with master...
I20260926 14:42:40.823046 591 heartbeater.cc:511] Master 127.16.90.254:38477 requested a full tablet report, sending...
I20260926 14:42:40.824013 32729 ts_manager.cc:194] Registered new tserver with Master: 6989f5b480a74ca4a112ce1d3e96b20c (127.16.90.194:43005)
I20260926 14:42:40.824563 32729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:33067
W20260926 14:42:40.918669 594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:42:40.918901 594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:42:40.918943 594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:42:40.922821 594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:42:40.922870 594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:42:40.922932 594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:42:40.927974 594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:33069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=33095
--tserver_master_addrs=127.16.90.254:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:42:40.928912 594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:42:40.929857 594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:42:40.939424 594 server_base.cc:1034] running on GCE node
W20260926 14:42:40.939364 602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:40.939308 599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:42:40.939312 600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:42:40.940038 594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:42:40.940624 594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:42:40.941789 594 hybrid_clock.cc:648] HybridClock initialized: now 1790433760941761 us; error 42 us; skew 500 ppm
Sep 26 14:42:40 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433760, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:42:40.944398 594 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:42:40.945257 594 webserver.cc:460] Webserver started at http://127.16.90.195:33095/ using document root <none> and password file <none>
I20260926 14:42:40.945782 594 fs_manager.cc:359] Metadata directory not provided
I20260926 14:42:40.945863 594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:42:40.949821 594 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 14:42:40.951766 609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:40.953089 594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:42:40.953208 594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "1953717dba5943ce79281d05ec1a85c0"
server_key_iv: "74dff83f38fb91ca62bbe46ce9e6ea28"
server_key_version: "encryptionkey@0"
I20260926 14:42:40.953728 594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:42:40.973501 594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:42:40.976403 594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:42:40.976574 594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:42:40.977180 594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:42:40.977973 594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:42:40.978018 594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:40.978060 594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:42:40.978089 594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:42:40.986464 594 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:33069
I20260926 14:42:40.986518 723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:33069 every 8 connection(s)
I20260926 14:42:40.987460 594 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:42:40 dist-test-slave-x4sn krb5kdc[32147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433760, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:42:40.997274 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 594
I20260926 14:42:41.001083 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:40.989214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:51827 (local address 127.16.90.254:38477)
0926 14:42:40.989351 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:40.989355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:40.990197 (+ 842us) server_negotiation.cc:407] Connection header received
0926 14:42:40.991493 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:40.991496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:40.991576 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:40.991652 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:40.992856 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.993379 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.994528 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.994747 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.994837 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:40.997258 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:40.997262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:40.997265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:40.997295 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:40.998808 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:40.999279 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:40.999282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:40.999284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:40.999367 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:40.999779 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:40.999783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:40.999784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:40.999988 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:41.000082 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:41.000678 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 14:42:41.000776 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:42:41.002063 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:40.989581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:51827)
0926 14:42:40.990046 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:40.990091 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:40.991284 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:40.991853 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:40.991860 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:40.992228 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:42:40.992713 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:40.992727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.993496 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.993499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:40.994401 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:40.994411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:40.994888 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:40.994892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:40.995056 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:40.995636 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:42:40.995652 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:42:40.997070 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:42:40.998916 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:40.998921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:40.998933 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:40.999189 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:40.999480 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:42:40.999484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:42:40.999486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:42:40.999657 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:42:41.000084 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:42:41.000090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:42:41.000379 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 14:42:41.000618 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 14:42:41.000986 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":106,"threads_started":1}
I20260926 14:42:41.003326 724 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38477
I20260926 14:42:41.003561 724 heartbeater.cc:464] Registering TS with master...
I20260926 14:42:41.004148 724 heartbeater.cc:511] Master 127.16.90.254:38477 requested a full tablet report, sending...
I20260926 14:42:41.005236 32729 ts_manager.cc:194] Registered new tserver with Master: 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069)
I20260926 14:42:41.005769 32729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:51827
I20260926 14:42:41.006438 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:40.997961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.16.90.254:38477)
0926 14:42:40.998235 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:40.998240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:40.998444 (+ 204us) server_negotiation.cc:407] Connection header received
0926 14:42:40.998555 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:40.998558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:40.998608 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:40.998708 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:40.999447 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.000145 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.001299 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.001638 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.001765 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:41.003456 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:41.003459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:41.003461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:41.003480 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:41.004781 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:41.005175 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:41.005177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:41.005178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:41.005256 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:41.005520 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:41.005524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:41.005526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:41.005694 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:41.005888 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:41.006176 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 14:42:41.006291 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
I20260926 14:42:41.008041 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:42:41.012008 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.008661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.16.90.254:38477)
0926 14:42:41.008779 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:41.008783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:41.008911 (+ 128us) server_negotiation.cc:407] Connection header received
0926 14:42:41.009066 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:41.009069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:41.009155 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:41.009238 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:42:41.009452 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.009954 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.010784 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.011015 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.011099 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:41.011384 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:42:41.011504 (+ 120us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 14:42:41.011892 (+ 388us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51664: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 14:42:41.012069 727 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51664: token expired
W20260926 14:42:41.012859 32658 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.90.254:38477)
I20260926 14:42:41.020169 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.013227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.16.90.254:38477)
0926 14:42:41.013926 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:41.013930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:41.013943 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:42:41.013976 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:41.013978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:41.014016 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:41.014086 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:42:41.014652 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.015198 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.016023 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.016236 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.016317 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:41.017532 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:42:41.017536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:42:41.017539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:42:41.017563 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:42:41.018742 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:41.019097 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:41.019101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:41.019102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:41.019178 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:42:41.019408 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:42:41.019411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:42:41.019413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:42:41.019574 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:42:41.019719 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:41.019924 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 14:42:41.020030 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620}
I20260926 14:42:41.022436 32729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:42:41.025669 32729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:42:41.033257 385 tablet_service.cc:1467] Processing CreateTablet for tablet d0f182310a6a43dcb4dd75f00a5234ae (DEFAULT_TABLE table=test-table [id=f7a32997eef94c81b5d2e4b155c3afbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:42:41.033913 385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f182310a6a43dcb4dd75f00a5234ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:41.036633 343 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: Bootstrap starting.
I20260926 14:42:41.037380 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.032093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33069 (local address 127.0.0.1:48978)
0926 14:42:41.032246 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:41.032261 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:41.032337 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:41.033427 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:41.033430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:41.033451 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:41.033567 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:41.033574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.034826 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.034830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:41.035939 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:41.035951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.036915 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.036919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:41.037040 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:41.037098 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 14:42:41.037161 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:42:41.037243 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 14:42:41.037599 343 tablet_bootstrap.cc:654] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:41.038048 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.032380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:43005 (local address 127.0.0.1:49584)
0926 14:42:41.032621 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:41.032633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:41.032730 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:41.033736 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:41.033738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:41.033754 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:41.033824 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:41.033829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.035542 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.035546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:41.036620 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:41.036631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.037645 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.037648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:41.037740 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:41.037761 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:42:41.037848 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 14:42:41.037918 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":123,"threads_started":1}
I20260926 14:42:41.038101 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.032336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.16.90.195:33069)
0926 14:42:41.032912 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:41.032924 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:41.032947 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:42:41.033021 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:41.033027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:41.033258 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:41.033540 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:41.033713 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.034725 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.036070 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.036753 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.037577 (+ 824us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:41.037742 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:41.037804 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 14:42:41.037920 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":162,"threads_started":1}
I20260926 14:42:41.038446 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.032809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.16.90.194:43005)
0926 14:42:41.033112 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:41.033125 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:41.033149 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:42:41.033219 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:41.033223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:41.033383 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:41.033552 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:41.033942 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.035411 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.036749 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.037550 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.037645 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:41.037783 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:41.038182 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 14:42:41.038289 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":98,"threads_started":1}
I20260926 14:42:41.039521 343 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: No bootstrap required, opened a new log
I20260926 14:42:41.039606 343 ts_tablet_manager.cc:1397] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:42:41.040026 343 raft_consensus.cc:348] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:41.040127 343 raft_consensus.cc:374] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:41.040161 343 raft_consensus.cc:729] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1228026955334c0ca1abd5e20e62c314, State: Initialized, Role: FOLLOWER
I20260926 14:42:41.040287 343 consensus_queue.cc:260] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:41.040580 343 ts_tablet_manager.cc:1428] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:42:41.041275 657 tablet_service.cc:1467] Processing CreateTablet for tablet d0f182310a6a43dcb4dd75f00a5234ae (DEFAULT_TABLE table=test-table [id=f7a32997eef94c81b5d2e4b155c3afbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:42:41.041683 524 tablet_service.cc:1467] Processing CreateTablet for tablet d0f182310a6a43dcb4dd75f00a5234ae (DEFAULT_TABLE table=test-table [id=f7a32997eef94c81b5d2e4b155c3afbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:42:41.042541 657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f182310a6a43dcb4dd75f00a5234ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:41.042830 524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f182310a6a43dcb4dd75f00a5234ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:42:41.051488 737 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Bootstrap starting.
I20260926 14:42:41.052229 738 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: Bootstrap starting.
I20260926 14:42:41.053510 737 tablet_bootstrap.cc:654] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:41.054497 738 tablet_bootstrap.cc:654] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: Neither blocks nor log segments found. Creating new log.
I20260926 14:42:41.054546 737 log.cc:824] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:41.055297 738 log.cc:824] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:42:41.056671 737 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: No bootstrap required, opened a new log
I20260926 14:42:41.056878 737 ts_tablet_manager.cc:1397] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:42:41.057440 738 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: No bootstrap required, opened a new log
I20260926 14:42:41.057641 738 ts_tablet_manager.cc:1397] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:42:41.059840 737 raft_consensus.cc:348] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:41.060081 737 raft_consensus.cc:374] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:41.060139 737 raft_consensus.cc:729] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1183f6d0fb5c40c4ae69912a73c5ea9e, State: Initialized, Role: FOLLOWER
I20260926 14:42:41.060544 737 consensus_queue.cc:260] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:41.060786 738 raft_consensus.cc:348] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:41.061017 738 raft_consensus.cc:374] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:42:41.061074 738 raft_consensus.cc:729] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6989f5b480a74ca4a112ce1d3e96b20c, State: Initialized, Role: FOLLOWER
I20260926 14:42:41.061146 724 heartbeater.cc:503] Master 127.16.90.254:38477 was elected leader, sending a full tablet report...
I20260926 14:42:41.061532 737 ts_tablet_manager.cc:1428] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:42:41.061553 738 consensus_queue.cc:260] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:41.062259 591 heartbeater.cc:503] Master 127.16.90.254:38477 was elected leader, sending a full tablet report...
I20260926 14:42:41.062516 738 ts_tablet_manager.cc:1428] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260926 14:42:41.105237 453 tablet.cc:2369] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:42:41.109463 458 raft_consensus.cc:482] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:42:41.109678 458 raft_consensus.cc:504] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:41.110831 458 leader_election.cc:288] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069), 6989f5b480a74ca4a112ce1d3e96b20c (127.16.90.194:43005)
I20260926 14:42:41.114475 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.111111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:50809 (local address 127.16.90.195:33069)
0926 14:42:41.111231 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:41.111237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:41.111300 (+ 63us) server_negotiation.cc:407] Connection header received
0926 14:42:41.111500 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:41.111504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:41.111560 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:41.111655 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:41.111949 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.112482 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.113546 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.113924 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.114016 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:41.114069 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:41.114286 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 14:42:41.114349 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 14:42:41.114481 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.111331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:38743 (local address 127.16.90.194:43005)
0926 14:42:41.111473 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:41.111477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:41.111553 (+ 76us) server_negotiation.cc:407] Connection header received
0926 14:42:41.111694 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:41.111697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:41.111743 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:41.111814 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:42:41.112072 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.112534 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.113557 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.113952 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.114057 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:41.114104 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:41.114307 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 14:42:41.114374 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:42:41.114482 454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.110997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33069 (local address 127.16.90.193:50809)
0926 14:42:41.111220 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:41.111235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:41.111337 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:41.111693 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:41.111696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:41.111714 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:41.111837 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:41.111844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.112584 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.112587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:41.113430 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:41.113439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.114031 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.114034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:41.114134 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:41.114189 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 14:42:41.114247 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 14:42:41.114320 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":10}
I20260926 14:42:41.114492 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.111064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:43005 (local address 127.16.90.193:38743)
0926 14:42:41.111469 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:41.111482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:42:41.111580 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:42:41.111843 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:42:41.111845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:42:41.111860 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:42:41.111971 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:41.111978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.112639 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.112642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:41.113437 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:42:41.113446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.114093 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.114098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:42:41.114169 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:41.114209 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 14:42:41.114279 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 14:42:41.114347 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":32,"thread_start_us":100,"threads_started":1}
I20260926 14:42:41.115118 677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f182310a6a43dcb4dd75f00a5234ae" candidate_uuid: "1228026955334c0ca1abd5e20e62c314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" is_pre_election: true
I20260926 14:42:41.115123 545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f182310a6a43dcb4dd75f00a5234ae" candidate_uuid: "1228026955334c0ca1abd5e20e62c314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" is_pre_election: true
I20260926 14:42:41.115375 677 raft_consensus.cc:2454] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1228026955334c0ca1abd5e20e62c314 in term 0.
I20260926 14:42:41.115382 545 raft_consensus.cc:2454] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1228026955334c0ca1abd5e20e62c314 in term 0.
I20260926 14:42:41.115832 337 leader_election.cc:302] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1228026955334c0ca1abd5e20e62c314, 6989f5b480a74ca4a112ce1d3e96b20c; no voters:
I20260926 14:42:41.116078 458 raft_consensus.cc:2781] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:42:41.116151 458 raft_consensus.cc:482] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:42:41.116189 458 raft_consensus.cc:3037] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:41.117223 458 raft_consensus.cc:504] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:41.117545 458 leader_election.cc:288] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 1 election: Requested vote from peers 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069), 6989f5b480a74ca4a112ce1d3e96b20c (127.16.90.194:43005)
I20260926 14:42:41.117853 545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f182310a6a43dcb4dd75f00a5234ae" candidate_uuid: "1228026955334c0ca1abd5e20e62c314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6989f5b480a74ca4a112ce1d3e96b20c"
I20260926 14:42:41.117853 677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f182310a6a43dcb4dd75f00a5234ae" candidate_uuid: "1228026955334c0ca1abd5e20e62c314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e"
I20260926 14:42:41.117972 677 raft_consensus.cc:3037] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:41.117995 545 raft_consensus.cc:3037] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:42:41.119218 545 raft_consensus.cc:2454] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1228026955334c0ca1abd5e20e62c314 in term 1.
I20260926 14:42:41.119218 677 raft_consensus.cc:2454] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1228026955334c0ca1abd5e20e62c314 in term 1.
I20260926 14:42:41.119482 337 leader_election.cc:302] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1228026955334c0ca1abd5e20e62c314, 6989f5b480a74ca4a112ce1d3e96b20c; no voters:
I20260926 14:42:41.119632 458 raft_consensus.cc:2781] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:42:41.119881 458 raft_consensus.cc:686] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 1 LEADER]: Becoming Leader. State: Replica: 1228026955334c0ca1abd5e20e62c314, State: Running, Role: LEADER
I20260926 14:42:41.120110 458 consensus_queue.cc:237] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:42:41.122975 32730 catalog_manager.cc:5495] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1228026955334c0ca1abd5e20e62c314 (127.16.90.193). New cstate: current_term: 1 leader_uuid: "1228026955334c0ca1abd5e20e62c314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } health_report { overall_health: HEALTHY } } }
I20260926 14:42:41.136972 460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.133744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.16.90.193:46765)
0926 14:42:41.133849 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:42:41.133852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:42:41.133864 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:42:41.133980 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:42:41.133982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:42:41.134031 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:42:41.134144 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:42:41.134313 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.134820 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.135563 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:42:41.135861 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:42:41.135948 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:42:41.136196 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:42:41.136591 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:42:41.136691 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:42:41.136786 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 14:42:41.136840 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:42:41.140404 677 raft_consensus.cc:1261] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 1 FOLLOWER]: Refusing update from remote peer 1228026955334c0ca1abd5e20e62c314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:42:41.140406 545 raft_consensus.cc:1261] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 1 FOLLOWER]: Refusing update from remote peer 1228026955334c0ca1abd5e20e62c314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:42:41.141043 458 consensus_queue.cc:1035] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:42:41.141228 744 consensus_queue.cc:1035] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:42:41.148476 747 mvcc.cc:205] Tried to move back new op lower bound from 7333616685625462784 to 7333616685550477312. Current Snapshot: MvccSnapshot[applied={T|T < 7333616685625462784}]
I20260926 14:42:41.149183 748 mvcc.cc:205] Tried to move back new op lower bound from 7333616685625462784 to 7333616685550477312. Current Snapshot: MvccSnapshot[applied={T|T < 7333616685625462784}]
I20260926 14:42:41.155771 32730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:42:41.155934 32730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:42:41.158032 32730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Sending DeleteTablet for 3 replicas of tablet d0f182310a6a43dcb4dd75f00a5234ae
I20260926 14:42:41.158643 385 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f182310a6a43dcb4dd75f00a5234ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:41 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:33954
I20260926 14:42:41.158783 657 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f182310a6a43dcb4dd75f00a5234ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:41 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:48978
I20260926 14:42:41.159060 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 300
I20260926 14:42:41.159276 524 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f182310a6a43dcb4dd75f00a5234ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:42:41 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:49584
I20260926 14:42:41.159941 757 tablet_replica.cc:330] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: stopping tablet replica
I20260926 14:42:41.160243 757 raft_consensus.cc:2229] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:42:41.160538 757 raft_consensus.cc:2258] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:42:41.161798 757 ts_tablet_manager.cc:1905] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:42:41.163923 757 ts_tablet_manager.cc:1918] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:42:41.164009 757 log.cc:1196] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/d0f182310a6a43dcb4dd75f00a5234ae
I20260926 14:42:41.164276 757 ts_tablet_manager.cc:1939] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Deleting consensus metadata
I20260926 14:42:41.165410 32715 catalog_manager.cc:4841] TS 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069): tablet d0f182310a6a43dcb4dd75f00a5234ae (table test-table [id=f7a32997eef94c81b5d2e4b155c3afbe]) successfully deleted
W20260926 14:42:41.166070 32716 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:46765 (error 108)
I20260926 14:42:41.166338 758 tablet_replica.cc:330] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: stopping tablet replica
I20260926 14:42:41.166699 758 raft_consensus.cc:2229] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:42:41.167043 758 raft_consensus.cc:2258] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:42:41.167147 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.166912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:46765 (local address 127.0.0.1:33982)
0926 14:42:41.167012 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:41.167070 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 14:42:41.167446 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 457
W20260926 14:42:41.167309 32716 catalog_manager.cc:4584] TS 1228026955334c0ca1abd5e20e62c314 (127.16.90.193:46765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0f182310a6a43dcb4dd75f00a5234ae: Network error: Client connection negotiation failed: client connection to 127.16.90.193:46765: connect: Connection refused (error 111)
I20260926 14:42:41.173382 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 594
I20260926 14:42:41.174726 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:42:41.173431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:43005 (local address 127.0.0.1:49598)
0926 14:42:41.174576 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0926 14:42:41.174632 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.194:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1065}
I20260926 14:42:41.178244 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 32697
I20260926 14:43:01.184021 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:38477
--webserver_interface=127.16.90.254
--webserver_port=43943
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:38477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:01.289436 760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:01.289695 760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:01.289748 760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:01.293686 760 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 14:43:01.293740 760 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 14:43:01.293761 760 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 14:43:01.293777 760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:01.293802 760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:01.293833 760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:01.293862 760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:01.293891 760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:01.299070 760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:38477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=43943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:01.300105 760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:01.301067 760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:01.309082 766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:01.309070 765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:01.309170 768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:01.309259 760 server_base.cc:1034] running on GCE node
I20260926 14:43:01.309926 760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:01.310953 760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:01.312103 760 hybrid_clock.cc:648] HybridClock initialized: now 1790433781312091 us; error 24 us; skew 500 ppm
Sep 26 14:43:01 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433781, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:01.320097 760 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:43:01.320971 760 webserver.cc:460] Webserver started at http://127.16.90.254:43943/ using document root <none> and password file <none>
I20260926 14:43:01.321507 760 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:01.321568 760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:01.325256 760 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 14:43:01.327255 775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:01.328441 760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260926 14:43:01.328540 760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "fc844351c82d4bc8bfa051d1eb47402a"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "aefcafd4328e43885d9967ecd1245c67"
server_key_iv: "5285ffacaa00333e5c70640a8cd330cd"
server_key_version: "encryptionkey@0"
I20260926 14:43:01.329082 760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:01.354487 760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:01.357355 760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:01.357506 760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:01.364308 760 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:38477
I20260926 14:43:01.364553 827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:38477 every 8 connection(s)
I20260926 14:43:01.365341 760 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:01.369414 828 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:43:01.369808 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 760
I20260926 14:43:01.370216 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:46765
--local_ip_for_outbound_sockets=127.16.90.193
--tserver_master_addrs=127.16.90.254:38477
--webserver_port=39373
--webserver_interface=127.16.90.193
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:43:01.371088 828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Bootstrap starting.
I20260926 14:43:01.394608 828 log.cc:824] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:01.408661 828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:43:01.409081 828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Bootstrap complete.
I20260926 14:43:01.414345 828 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } }
I20260926 14:43:01.414844 828 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc844351c82d4bc8bfa051d1eb47402a, State: Initialized, Role: FOLLOWER
I20260926 14:43:01.415376 828 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } }
I20260926 14:43:01.415515 828 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:01.415589 828 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:01.415707 828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 2 FOLLOWER]: Advancing to term 3
I20260926 14:43:01.418032 828 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } }
I20260926 14:43:01.418380 828 leader_election.cc:302] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc844351c82d4bc8bfa051d1eb47402a; no voters:
I20260926 14:43:01.418749 828 leader_election.cc:288] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 14:43:01.418851 832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 3 FOLLOWER]: Leader election won for term 3
I20260926 14:43:01.419098 832 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [term 3 LEADER]: Becoming Leader. State: Replica: fc844351c82d4bc8bfa051d1eb47402a, State: Running, Role: LEADER
I20260926 14:43:01.419435 832 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } }
I20260926 14:43:01.420181 828 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:01.420868 834 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc844351c82d4bc8bfa051d1eb47402a. Latest consensus state: current_term: 3 leader_uuid: "fc844351c82d4bc8bfa051d1eb47402a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } } }
I20260926 14:43:01.420975 834 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:01.421476 833 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fc844351c82d4bc8bfa051d1eb47402a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc844351c82d4bc8bfa051d1eb47402a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38477 } } }
I20260926 14:43:01.421571 833 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:01.422385 840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:01.427326 840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:01.428010 840 catalog_manager.cc:1259] Loaded cluster ID: 7281af1ddd674f93bc2bc6bfdf071722
I20260926 14:43:01.428069 840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:01.430567 840 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:01.431252 840 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Loaded TSK: 0
I20260926 14:43:01.431330 840 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc844351c82d4bc8bfa051d1eb47402a: Loaded TSK: 1
F20260926 14:43:01.432087 840 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f79210829ad google::LogMessage::Fail() at ??:0
@ 0x7f792108588a google::LogMessage::SendToLog() at ??:0
@ 0x7f79210824de google::LogMessage::Flush() at ??:0
@ 0x7f7921083a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7923d4618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7923d3ef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f792b9e77bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f792b9c199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f792b9c2178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f792ba03df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f792ba2282b std::function<>::operator()() at ??:0
@ 0x7f792b9c1b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f792b9c2f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f792b9bf579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f792ba03640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f79232d43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f79232d4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f79232d63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f79232c5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f79249096db start_thread at ??:0
@ 0x7f791fb8061f clone at ??:0
W20260926 14:43:01.526470 830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:01.526715 830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:01.526760 830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:01.530803 830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:01.530864 830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:01.530925 830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:43:01.536219 830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:46765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=39373
--tserver_master_addrs=127.16.90.254:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:01.537293 830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:01.538300 830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:01.547344 876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:01.547456 877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:01.547399 879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:01.547780 830 server_base.cc:1034] running on GCE node
I20260926 14:43:01.548149 830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:01.548725 830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:01.549893 830 hybrid_clock.cc:648] HybridClock initialized: now 1790433781549874 us; error 32 us; skew 500 ppm
Sep 26 14:43:01 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433781, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:01.552518 830 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:43:01.553385 830 webserver.cc:460] Webserver started at http://127.16.90.193:39373/ using document root <none> and password file <none>
I20260926 14:43:01.553884 830 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:01.553947 830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:01.557593 830 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:01.559559 886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:01.560823 830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:01.560912 830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "1228026955334c0ca1abd5e20e62c314"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "bff2e731336969b7a9bb95d4dd72b19d"
server_key_iv: "161e9a6665e2781cab401e42e769e500"
server_key_version: "encryptionkey@0"
I20260926 14:43:01.561503 830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:01.573433 830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:01.576306 830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:01.576475 830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:01.577023 830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:01.578145 893 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:43:01.583235 830 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:43:01.583289 830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 14:43:01.583359 830 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:43:01.584934 830 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:43:01.584980 830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 14:43:01.585290 893 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: Bootstrap starting.
I20260926 14:43:01.594859 830 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:46765
I20260926 14:43:01.595116 1000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:46765 every 8 connection(s)
I20260926 14:43:01.595885 830 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:43:01.597554 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 830
I20260926 14:43:01.598021 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:43005
--local_ip_for_outbound_sockets=127.16.90.194
--tserver_master_addrs=127.16.90.254:38477
--webserver_port=46033
--webserver_interface=127.16.90.194
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:43:01.607746 893 log.cc:824] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:01.614305 893 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:43:01.614632 893 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: Bootstrap complete.
I20260926 14:43:01.615532 893 ts_tablet_manager.cc:1397] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.004s
I20260926 14:43:01.617803 893 raft_consensus.cc:348] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:43:01.618201 893 raft_consensus.cc:729] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1228026955334c0ca1abd5e20e62c314, State: Initialized, Role: FOLLOWER
I20260926 14:43:01.618641 893 consensus_queue.cc:260] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:43:01.619468 893 ts_tablet_manager.cc:1428] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260926 14:43:01.649148 1003 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:01.597877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:36363)
0926 14:43:01.599232 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:01.599290 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:01.600304 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:01.648403 (+ 48099us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1123,"thread_start_us":146,"threads_started":1}
W20260926 14:43:01.649729 1001 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:38477 (0 consecutive failures): Network error: Failed to ping master at 127.16.90.254:38477: Client connection negotiation failed: client connection to 127.16.90.254:38477: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:43:01.650511 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:01.650238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:47163)
0926 14:43:01.650370 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:01.650446 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 14:43:01.651149 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:01.650942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:45049)
0926 14:43:01.651042 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:01.651092 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 14:43:01.651300 1001 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:43:01.723151 1004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:01.723397 1004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:01.723438 1004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:01.727332 1004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:01.727383 1004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:01.727439 1004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:43:01.732489 1004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:43005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=46033
--tserver_master_addrs=127.16.90.254:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:01.733422 1004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:01.734337 1004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:43:01.743086 1004 server_base.cc:1034] running on GCE node
W20260926 14:43:01.743047 1012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:01.743029 1011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:01.743048 1014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:01.743602 1004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:01.744259 1004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:01.745415 1004 hybrid_clock.cc:648] HybridClock initialized: now 1790433781745399 us; error 28 us; skew 500 ppm
Sep 26 14:43:01 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433781, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:01.747903 1004 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:43:01.748692 1004 webserver.cc:460] Webserver started at http://127.16.90.194:46033/ using document root <none> and password file <none>
I20260926 14:43:01.749193 1004 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:01.749258 1004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:01.753028 1004 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:01.755187 1021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:01.756386 1004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:01.756490 1004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "6989f5b480a74ca4a112ce1d3e96b20c"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "cbbf87f38ea95c620cb1c311d525cd2e"
server_key_iv: "a63183783e5093c040fd18888f56989d"
server_key_version: "encryptionkey@0"
I20260926 14:43:01.757022 1004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:01.780189 1004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:01.783033 1004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:01.783226 1004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:01.783854 1004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:01.784956 1028 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:43:01.789093 1004 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:43:01.789158 1004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 14:43:01.789225 1004 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:43:01.790861 1004 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:43:01.790906 1004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 14:43:01.791195 1028 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: Bootstrap starting.
I20260926 14:43:01.800027 1004 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:43005
I20260926 14:43:01.800103 1135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:43005 every 8 connection(s)
I20260926 14:43:01.800935 1004 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:43:01.804114 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:01.802737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:38187)
0926 14:43:01.803208 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:01.803333 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
W20260926 14:43:01.804591 1136 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:38477 (0 consecutive failures): Network error: Failed to ping master at 127.16.90.254:38477: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
I20260926 14:43:01.805220 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:01.804971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:47461)
0926 14:43:01.805101 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:01.805158 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 14:43:01.805789 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:01.805590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:60299)
0926 14:43:01.805687 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:01.805735 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 14:43:01.805938 1136 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:43:01.808544 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 1004
I20260926 14:43:01.808996 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:33069
--local_ip_for_outbound_sockets=127.16.90.195
--tserver_master_addrs=127.16.90.254:38477
--webserver_port=33095
--webserver_interface=127.16.90.195
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:43:01.809533 1028 log.cc:824] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:01.816143 1028 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:43:01.816494 1028 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: Bootstrap complete.
I20260926 14:43:01.817812 1028 ts_tablet_manager.cc:1397] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.007s
I20260926 14:43:01.821441 1028 raft_consensus.cc:348] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:43:01.822005 1028 raft_consensus.cc:729] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6989f5b480a74ca4a112ce1d3e96b20c, State: Initialized, Role: FOLLOWER
I20260926 14:43:01.822527 1028 consensus_queue.cc:260] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:43:01.823585 1028 ts_tablet_manager.cc:1428] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260926 14:43:01.823613 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:01.823385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:55451)
0926 14:43:01.823500 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:01.823551 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 14:43:01.824292 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:01.824051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:49441)
0926 14:43:01.824169 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:01.824228 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 14:43:01.922336 1140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:01.922591 1140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:01.922633 1140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:01.926549 1140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:01.926613 1140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:01.926671 1140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:43:01.931928 1140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:33069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=33095
--tserver_master_addrs=127.16.90.254:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:01.932962 1140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:01.933920 1140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:01.943322 1146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:01.943375 1140 server_base.cc:1034] running on GCE node
W20260926 14:43:01.943315 1147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:01.943346 1149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:01.943974 1140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:01.944602 1140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:01.945761 1140 hybrid_clock.cc:648] HybridClock initialized: now 1790433781945738 us; error 32 us; skew 500 ppm
Sep 26 14:43:01 dist-test-slave-x4sn krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433781, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:01.948266 1140 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:43:01.949131 1140 webserver.cc:460] Webserver started at http://127.16.90.195:33095/ using document root <none> and password file <none>
I20260926 14:43:01.949664 1140 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:01.949731 1140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:01.953490 1140 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 14:43:01.955412 1156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:01.956606 1140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:01.956717 1140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e"
format_stamp: "Formatted at 2026-09-26 14:42:19 on dist-test-slave-x4sn"
server_key: "1953717dba5943ce79281d05ec1a85c0"
server_key_iv: "74dff83f38fb91ca62bbe46ce9e6ea28"
server_key_version: "encryptionkey@0"
I20260926 14:43:01.957230 1140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:01.994303 1140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:01.997192 1140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:01.997378 1140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:01.997987 1140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:01.998867 1140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:01.998914 1140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:01.998965 1140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:01.998994 1140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:02.007766 1140 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:33069
I20260926 14:43:02.007802 1269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:33069 every 8 connection(s)
I20260926 14:43:02.008795 1140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:43:02.010164 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 1140
I20260926 14:43:02.013149 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:02.010972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:53785)
0926 14:43:02.011914 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:02.012016 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":137,"threads_started":1}
W20260926 14:43:02.013603 1270 heartbeater.cc:644] Failed to heartbeat to 127.16.90.254:38477 (0 consecutive failures): Network error: Failed to ping master at 127.16.90.254:38477: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
I20260926 14:43:02.014153 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:02.013955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:38379)
0926 14:43:02.014054 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:02.014098 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 14:43:02.014679 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:02.014517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:37089)
0926 14:43:02.014591 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:02.014627 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 14:43:02.014827 1270 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:43:02.652318 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:02.651887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:38653)
0926 14:43:02.652141 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:02.652214 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260926 14:43:02.825325 1275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:02.824925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:38549)
0926 14:43:02.825154 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:02.825230 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260926 14:43:03.015936 1276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:03.015466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:44601)
0926 14:43:03.015772 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:03.015834 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20260926 14:43:03.159346 1277 raft_consensus.cc:482] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:43:03.159550 1277 raft_consensus.cc:504] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:43:03.160717 1277 leader_election.cc:288] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069), 6989f5b480a74ca4a112ce1d3e96b20c (127.16.90.194:43005)
Sep 26 14:43:03 dist-test-slave-x4sn krb5kdc[32147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433781, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.195@KRBTEST.COM
Sep 26 14:43:03 dist-test-slave-x4sn krb5kdc[32147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433781, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.194@KRBTEST.COM
I20260926 14:43:03.173331 1279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:03.160962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:43005 (local address 127.16.90.193:57015)
0926 14:43:03.161270 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:03.161283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:03.161381 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:03.162536 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:03.162542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:03.163064 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:03.163533 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:03.163548 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:03.164845 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:03.164848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:03.165898 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:03.165907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:03.166587 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:03.166591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:03.166770 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:03.167342 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:03.167363 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:03.169372 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:03.171204 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:03.171206 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:03.171208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:03.171434 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:03.171780 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:03.171782 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:03.171783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:03.171855 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:03.172416 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:03.172422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:03.172648 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 14:43:03.172806 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 14:43:03.173124 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":159,"spinlock_wait_cycles":16000,"thread_start_us":69,"threads_started":1}
I20260926 14:43:03.173341 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:03.161078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:36599 (local address 127.16.90.195:33069)
0926 14:43:03.161548 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:03.161585 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:03.161621 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:43:03.162291 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:03.162296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:03.162584 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:03.162832 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:03.163713 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:03.164746 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:03.166000 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:03.166434 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:03.166590 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:03.169236 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:03.169242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:03.169259 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:03.169291 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:03.170894 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:03.171441 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:03.171447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:03.171452 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:03.171581 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:03.171929 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:03.171933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:03.171934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:03.172306 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:03.172421 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:03.172927 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 14:43:03.173167 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":117,"threads_started":1}
I20260926 14:43:03.173338 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:03.161214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:57015 (local address 127.16.90.194:43005)
0926 14:43:03.161661 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:03.161671 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:03.161715 (+ 44us) server_negotiation.cc:407] Connection header received
0926 14:43:03.161894 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:03.161898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:03.162210 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:03.162479 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:03.163724 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:03.164717 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:03.166005 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:03.166473 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:03.166621 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:03.169509 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:03.169513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:03.169525 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:03.169551 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:03.171098 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:03.171538 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:03.171542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:03.171546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:03.171696 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:03.171939 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:03.171941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:03.171942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:03.172306 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:03.172421 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:03.172928 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 14:43:03.173167 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":80,"threads_started":1}
I20260926 14:43:03.173350 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:03.160905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33069 (local address 127.16.90.193:36599)
0926 14:43:03.161115 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:03.161128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:03.161230 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:03.162796 (+ 1566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:03.162799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:03.163097 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:03.163533 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:03.163548 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:03.164855 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:03.164859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:03.165895 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:03.165905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:03.166542 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:03.166546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:03.166764 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:03.167383 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:03.167395 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:03.169025 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:03.171010 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:03.171016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:03.171029 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:03.171346 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:03.171684 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:03.171688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:03.171690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:03.171844 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:03.172415 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:03.172421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:03.172651 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 14:43:03.172806 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 14:43:03.173194 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":290,"spinlock_wait_cycles":38144,"thread_start_us":71,"threads_started":1}
I20260926 14:43:03.174751 1090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f182310a6a43dcb4dd75f00a5234ae" candidate_uuid: "1228026955334c0ca1abd5e20e62c314" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" is_pre_election: true
I20260926 14:43:03.175034 1090 raft_consensus.cc:2454] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1228026955334c0ca1abd5e20e62c314 in term 1.
I20260926 14:43:03.175977 887 leader_election.cc:302] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1228026955334c0ca1abd5e20e62c314, 6989f5b480a74ca4a112ce1d3e96b20c; no voters:
I20260926 14:43:03.176224 1277 raft_consensus.cc:2781] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 14:43:03.176316 1277 raft_consensus.cc:482] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:43:03.176365 1277 raft_consensus.cc:3037] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 1 FOLLOWER]: Advancing to term 2
W20260926 14:43:03.176791 888 leader_election.cc:341] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069): Not found: Tablet not found: d0f182310a6a43dcb4dd75f00a5234ae
I20260926 14:43:03.178483 1277 raft_consensus.cc:504] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:43:03.178794 1277 leader_election.cc:288] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 2 election: Requested vote from peers 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069), 6989f5b480a74ca4a112ce1d3e96b20c (127.16.90.194:43005)
I20260926 14:43:03.179123 1090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f182310a6a43dcb4dd75f00a5234ae" candidate_uuid: "1228026955334c0ca1abd5e20e62c314" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6989f5b480a74ca4a112ce1d3e96b20c"
I20260926 14:43:03.179245 1090 raft_consensus.cc:3037] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 1 FOLLOWER]: Advancing to term 2
W20260926 14:43:03.179306 888 leader_election.cc:341] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069): Not found: Tablet not found: d0f182310a6a43dcb4dd75f00a5234ae
I20260926 14:43:03.181151 1090 raft_consensus.cc:2454] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1228026955334c0ca1abd5e20e62c314 in term 2.
I20260926 14:43:03.181412 887 leader_election.cc:302] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1228026955334c0ca1abd5e20e62c314, 6989f5b480a74ca4a112ce1d3e96b20c; no voters: 1183f6d0fb5c40c4ae69912a73c5ea9e
I20260926 14:43:03.181571 1277 raft_consensus.cc:2781] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:43:03.181761 1277 raft_consensus.cc:686] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [term 2 LEADER]: Becoming Leader. State: Replica: 1228026955334c0ca1abd5e20e62c314, State: Running, Role: LEADER
I20260926 14:43:03.182066 1277 consensus_queue.cc:237] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:43:03.182410 1279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:03.182175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:35339)
0926 14:43:03.182309 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:03.182350 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 14:43:03.791002 1090 raft_consensus.cc:1261] T d0f182310a6a43dcb4dd75f00a5234ae P 6989f5b480a74ca4a112ce1d3e96b20c [term 2 FOLLOWER]: Refusing update from remote peer 1228026955334c0ca1abd5e20e62c314: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260926 14:43:03.791766 1284 consensus_queue.cc:1035] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 14:43:03.793851 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:03.793470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:37401)
0926 14:43:03.793691 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:03.793752 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":48,"threads_started":1}
W20260926 14:43:03.795394 888 consensus_peers.cc:487] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314 -> Peer 1183f6d0fb5c40c4ae69912a73c5ea9e (127.16.90.195:33069): Couldn't send request to peer 1183f6d0fb5c40c4ae69912a73c5ea9e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0f182310a6a43dcb4dd75f00a5234ae. This is attempt 1: this message will repeat every 5th retry.
I20260926 14:43:03.796564 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:03.796209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:54805)
0926 14:43:03.796401 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:03.796457 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":53,"threads_started":1}
I20260926 14:43:03.796885 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:03.796642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:47037)
0926 14:43:03.796784 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:03.796829 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 14:43:04.017028 1292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:04.016597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:60291)
0926 14:43:04.016865 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:04.016927 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20260926 14:43:04.224102 1293 ts_tablet_manager.cc:927] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Initiating tablet copy from peer 1228026955334c0ca1abd5e20e62c314 (127.16.90.193:46765)
I20260926 14:43:04.224733 1293 tablet_copy_client.cc:316] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.90.193:46765
Sep 26 14:43:04 dist-test-slave-x4sn krb5kdc[32147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433781, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.193@KRBTEST.COM
I20260926 14:43:04.233930 1292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:04.225209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:46765 (local address 127.16.90.195:51013)
0926 14:43:04.225322 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:04.225345 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:04.225453 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:04.226070 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:04.226075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:04.226444 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:04.226562 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:04.226568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:04.227650 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:04.227652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:04.228334 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:04.228343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:04.228866 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:04.228868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:04.228938 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:04.229465 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:04.229477 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:04.230834 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:04.232601 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:04.232608 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:04.232613 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:04.232823 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:04.233042 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:04.233044 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:04.233045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:04.233119 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:04.233467 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:04.233472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:04.233571 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 14:43:04.233681 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 14:43:04.233801 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 14:43:04.233982 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:04.225325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:51013 (local address 127.16.90.193:46765)
0926 14:43:04.225683 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:04.225692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:04.225712 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:43:04.225752 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:04.225756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:04.225929 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:04.226037 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:04.226682 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:04.227553 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:04.228430 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:04.228778 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:04.228858 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:04.230954 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:04.230965 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:04.230981 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:04.231005 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:04.232506 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:04.232906 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:04.232910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:04.232911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:04.232981 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:04.233175 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:04.233177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:04.233178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:04.233391 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:04.233486 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:04.233751 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 14:43:04.233865 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":85,"threads_started":1}
I20260926 14:43:04.234954 975 tablet_copy_service.cc:140] P 1228026955334c0ca1abd5e20e62c314: Received BeginTabletCopySession request for tablet d0f182310a6a43dcb4dd75f00a5234ae from peer 1183f6d0fb5c40c4ae69912a73c5ea9e ({username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:51013)
I20260926 14:43:04.235105 975 tablet_copy_service.cc:161] P 1228026955334c0ca1abd5e20e62c314: Beginning new tablet copy session on tablet d0f182310a6a43dcb4dd75f00a5234ae from peer 1183f6d0fb5c40c4ae69912a73c5ea9e at {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:51013: session id = 1183f6d0fb5c40c4ae69912a73c5ea9e-d0f182310a6a43dcb4dd75f00a5234ae
I20260926 14:43:04.236217 975 tablet_copy_source_session.cc:205] T d0f182310a6a43dcb4dd75f00a5234ae P 1228026955334c0ca1abd5e20e62c314: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 14:43:04.239892 1293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f182310a6a43dcb4dd75f00a5234ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:04.246057 1293 tablet_copy_client.cc:791] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: tablet copy: Starting download of 0 data blocks...
I20260926 14:43:04.246227 1293 tablet_copy_client.cc:655] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: tablet copy: Starting download of 1 WAL segments...
I20260926 14:43:04.247756 1293 tablet_copy_client.cc:531] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 14:43:04.249325 1293 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Bootstrap starting.
I20260926 14:43:04.254277 1293 log.cc:824] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:04.260874 1293 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:43:04.261220 1293 tablet_bootstrap.cc:492] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Bootstrap complete.
I20260926 14:43:04.261521 1293 ts_tablet_manager.cc:1397] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.001s
I20260926 14:43:04.263823 1293 raft_consensus.cc:348] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:43:04.264088 1293 raft_consensus.cc:729] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1183f6d0fb5c40c4ae69912a73c5ea9e, State: Initialized, Role: FOLLOWER
I20260926 14:43:04.264489 1293 consensus_queue.cc:260] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183f6d0fb5c40c4ae69912a73c5ea9e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33069 } } peers { permanent_uuid: "6989f5b480a74ca4a112ce1d3e96b20c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43005 } } peers { permanent_uuid: "1228026955334c0ca1abd5e20e62c314" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 46765 } }
I20260926 14:43:04.265410 1293 ts_tablet_manager.cc:1428] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 14:43:04.265626 1292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:04.265368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:38025)
0926 14:43:04.265489 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:04.265550 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 14:43:04.266103 975 tablet_copy_service.cc:342] P 1228026955334c0ca1abd5e20e62c314: Request end of tablet copy session 1183f6d0fb5c40c4ae69912a73c5ea9e-d0f182310a6a43dcb4dd75f00a5234ae received from {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:51013
I20260926 14:43:04.266199 975 tablet_copy_service.cc:433] P 1228026955334c0ca1abd5e20e62c314: ending tablet copy session 1183f6d0fb5c40c4ae69912a73c5ea9e-d0f182310a6a43dcb4dd75f00a5234ae on tablet d0f182310a6a43dcb4dd75f00a5234ae with peer 1183f6d0fb5c40c4ae69912a73c5ea9e
I20260926 14:43:04.266269 1292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:04.266058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:54391)
0926 14:43:04.266158 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:04.266205 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:43:04.628289 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:04.627831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:34235)
0926 14:43:04.628116 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:04.628179 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":60,"threads_started":1}
I20260926 14:43:04.628346 1224 raft_consensus.cc:1203] T d0f182310a6a43dcb4dd75f00a5234ae P 1183f6d0fb5c40c4ae69912a73c5ea9e [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 14:43:04.629303 1292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:04.629071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:36715)
0926 14:43:04.629179 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:04.629235 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 14:43:04.797879 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:04.797504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:60965)
0926 14:43:04.797730 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:04.797784 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":63,"threads_started":1}
I20260926 14:43:05.629393 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:05.628985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:33971)
0926 14:43:05.629240 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:05.629299 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260926 14:43:05.630272 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:05.629872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:42469)
0926 14:43:05.630145 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:05.630197 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":66,"threads_started":1}
I20260926 14:43:05.799177 1304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:05.798574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:42255)
0926 14:43:05.798899 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:05.798985 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":84,"threads_started":1}
I20260926 14:43:06.630725 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:06.630301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:58757)
0926 14:43:06.630563 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:06.630616 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260926 14:43:06.631346 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:06.630907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:55313)
0926 14:43:06.631192 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:06.631254 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
I20260926 14:43:06.631484 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:06.631211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:43949)
0926 14:43:06.631356 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:06.631405 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 14:43:06.800446 1307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:06.800001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:53161)
0926 14:43:06.800292 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:06.800359 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260926 14:43:07.632608 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:07.632221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:46703)
0926 14:43:07.632453 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:07.632508 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
I20260926 14:43:07.632673 1309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:07.632207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:46383)
0926 14:43:07.632527 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:07.632579 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":89,"threads_started":1}
I20260926 14:43:07.801613 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:07.801250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:50683)
0926 14:43:07.801480 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:07.801531 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260926 14:43:08.633769 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:08.633352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:60751)
0926 14:43:08.633591 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:08.633653 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":78,"threads_started":1}
I20260926 14:43:08.633838 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:08.633368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:51481)
0926 14:43:08.633706 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:08.633759 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":80,"threads_started":1}
I20260926 14:43:08.802695 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:08.802309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:51383)
0926 14:43:08.802553 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:08.802611 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260926 14:43:09.634974 1315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:09.634564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:45193)
0926 14:43:09.634818 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:09.634876 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260926 14:43:09.635032 1314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:09.634564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:32853)
0926 14:43:09.634895 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:09.634949 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":88,"threads_started":1}
I20260926 14:43:09.803912 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:09.803462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:34267)
0926 14:43:09.803767 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:09.803824 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
I20260926 14:43:10.636137 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:10.635741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:39597)
0926 14:43:10.635985 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:10.636041 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
I20260926 14:43:10.636168 1318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:10.635790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:42663)
0926 14:43:10.636032 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:10.636084 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260926 14:43:10.805012 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:10.804621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:53257)
0926 14:43:10.804867 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:10.804925 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
I20260926 14:43:11.637409 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:11.636985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:52753)
0926 14:43:11.637258 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:11.637316 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":94,"threads_started":1}
I20260926 14:43:11.637465 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:11.636985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:48691)
0926 14:43:11.637326 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:11.637380 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":97,"threads_started":1}
I20260926 14:43:11.806116 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:11.805725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:56671)
0926 14:43:11.805976 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:11.806033 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260926 14:43:12.638545 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:12.638141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:36087)
0926 14:43:12.638395 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:12.638452 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260926 14:43:12.638576 1324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:12.638193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:38911)
0926 14:43:12.638440 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:12.638493 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260926 14:43:12.807188 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:12.806831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:45761)
0926 14:43:12.807052 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:12.807106 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
I20260926 14:43:13.639652 1326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:13.639252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:53765)
0926 14:43:13.639500 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:13.639555 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260926 14:43:13.639693 1327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:13.639285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:59413)
0926 14:43:13.639517 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:13.639569 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260926 14:43:13.808301 1328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:13.807896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:58965)
0926 14:43:13.808151 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:13.808210 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
I20260926 14:43:14.640899 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:14.640460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:46479)
0926 14:43:14.640737 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:14.640800 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
I20260926 14:43:14.640942 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:14.640460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:33979)
0926 14:43:14.640808 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:14.640860 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":95,"threads_started":1}
I20260926 14:43:14.809514 1331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:14.809094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:41669)
0926 14:43:14.809359 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:14.809418 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
I20260926 14:43:15.642015 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:15.641621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:55321)
0926 14:43:15.641867 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:15.641920 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
I20260926 14:43:15.642086 1333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:15.641621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:48053)
0926 14:43:15.641955 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:15.642005 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":96,"threads_started":1}
I20260926 14:43:15.810621 1334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:15.810232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:55135)
0926 14:43:15.810472 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:15.810531 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260926 14:43:16.643236 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:16.642836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.195:47205)
0926 14:43:16.643069 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:16.643124 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260926 14:43:16.643301 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:16.642922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.193:52089)
0926 14:43:16.643165 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:16.643214 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
I20260926 14:43:16.811937 1337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:16.811485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38477 (local address 127.16.90.194:42941)
0926 14:43:16.811775 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:16.811836 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.254:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260926 14:43:17.021392 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 830
I20260926 14:43:17.026809 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 1004
I20260926 14:43:17.031757 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 1140
2026-09-26T14:43:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59802 ms)
[----------] 30 tests from SecurityITest (235398 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:17 dist-test-slave-x4sn krb5kdc[1340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:17 dist-test-slave-x4sn krb5kdc[1340](info): set up 2 sockets
Sep 26 14:43:17 dist-test-slave-x4sn krb5kdc[1340](info): commencing operation
krb5kdc: starting...
W20260926 14:43:18.773204 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:18 dist-test-slave-x4sn krb5kdc[1340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:43:18.889808 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:38279
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:38279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:18.997531 1353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:18.997818 1353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:18.997875 1353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:19.001946 1353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:19.002034 1353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:19.002061 1353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:19.002082 1353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:19.002103 1353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:19.007547 1353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:38279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:38279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:19.008690 1353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:19.009727 1353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:19.018682 1359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:19.018702 1353 server_base.cc:1034] running on GCE node
W20260926 14:43:19.018674 1358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:19.018667 1361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:19.019536 1353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:19.020746 1353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:19.021909 1353 hybrid_clock.cc:648] HybridClock initialized: now 1790433799021892 us; error 34 us; skew 500 ppm
Sep 26 14:43:19 dist-test-slave-x4sn krb5kdc[1340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433799, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:19.024600 1353 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:43:19.025509 1353 webserver.cc:460] Webserver started at http://127.16.90.254:41077/ using document root <none> and password file <none>
I20260926 14:43:19.026039 1353 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:19.026103 1353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:19.026306 1353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:19.028363 1353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "aa155a73e3ad48179dbd49fb30d58bcb"
format_stamp: "Formatted at 2026-09-26 14:43:19 on dist-test-slave-x4sn"
server_key: "9d0fa303a10117cd7d7017fb1796a3aa"
server_key_iv: "06d6c3d3517c9188c63cf93c6aff09af"
server_key_version: "encryptionkey@0"
I20260926 14:43:19.028893 1353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "aa155a73e3ad48179dbd49fb30d58bcb"
format_stamp: "Formatted at 2026-09-26 14:43:19 on dist-test-slave-x4sn"
server_key: "9d0fa303a10117cd7d7017fb1796a3aa"
server_key_iv: "06d6c3d3517c9188c63cf93c6aff09af"
server_key_version: "encryptionkey@0"
I20260926 14:43:19.032456 1353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:43:19.034787 1368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:19.036019 1353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:19.036128 1353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "aa155a73e3ad48179dbd49fb30d58bcb"
format_stamp: "Formatted at 2026-09-26 14:43:19 on dist-test-slave-x4sn"
server_key: "9d0fa303a10117cd7d7017fb1796a3aa"
server_key_iv: "06d6c3d3517c9188c63cf93c6aff09af"
server_key_version: "encryptionkey@0"
I20260926 14:43:19.036219 1353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:19.057257 1353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:19.060077 1353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:19.060256 1353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:19.066915 1353 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:38279
I20260926 14:43:19.066929 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:38279 every 8 connection(s)
I20260926 14:43:19.067845 1353 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:19.070428 1421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:19.075783 1421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb: Bootstrap starting.
I20260926 14:43:19.075891 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 1353
I20260926 14:43:19.076006 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:19.076284 16747 external_mini_cluster.cc:1428] Setting key b72589298b2b3de7575a3dd13dbc8980
I20260926 14:43:19.078323 1421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:19.079182 1421 log.cc:824] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:19.081310 1421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb: No bootstrap required, opened a new log
I20260926 14:43:19.085243 1421 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa155a73e3ad48179dbd49fb30d58bcb" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38279 } }
Sep 26 14:43:19 dist-test-slave-x4sn krb5kdc[1340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:19.085508 1421 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:19.085567 1421 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa155a73e3ad48179dbd49fb30d58bcb, State: Initialized, Role: FOLLOWER
I20260926 14:43:19.086050 1421 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa155a73e3ad48179dbd49fb30d58bcb" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38279 } }
I20260926 14:43:19.086191 1421 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:19.086252 1421 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:19.086344 1421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:19.087293 1421 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa155a73e3ad48179dbd49fb30d58bcb" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38279 } }
I20260926 14:43:19.087605 1421 leader_election.cc:302] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa155a73e3ad48179dbd49fb30d58bcb; no voters:
I20260926 14:43:19.087929 1421 leader_election.cc:288] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:19.088018 1426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:19.088254 1426 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [term 1 LEADER]: Becoming Leader. State: Replica: aa155a73e3ad48179dbd49fb30d58bcb, State: Running, Role: LEADER
I20260926 14:43:19.088617 1426 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa155a73e3ad48179dbd49fb30d58bcb" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38279 } }
I20260926 14:43:19.089268 1421 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:19.089798 1426 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa155a73e3ad48179dbd49fb30d58bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa155a73e3ad48179dbd49fb30d58bcb" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38279 } } }
I20260926 14:43:19.089993 1426 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:19.090396 1433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:19.090572 1428 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa155a73e3ad48179dbd49fb30d58bcb. Latest consensus state: current_term: 1 leader_uuid: "aa155a73e3ad48179dbd49fb30d58bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa155a73e3ad48179dbd49fb30d58bcb" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 38279 } } }
I20260926 14:43:19.090756 1428 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:19.091073 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.077553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.16.90.254:38279)
0926 14:43:19.077971 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:19.077990 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:19.078030 (+ 40us) server_negotiation.cc:407] Connection header received
0926 14:43:19.078849 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:19.078872 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:19.079301 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:19.079662 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:19.080415 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.081878 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.082686 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.083376 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.083530 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.086074 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:19.086079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:19.086091 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:19.086116 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:19.087799 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:19.088255 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:19.088261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:19.088266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:19.088412 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:19.088700 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:19.088704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:19.088706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:19.089207 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:19.089441 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:19.089812 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 14:43:19.090112 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
I20260926 14:43:19.093914 1433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:19.099272 1433 catalog_manager.cc:1347] Generated new cluster ID: 7c0af9cd70f642dcb18e39b64bd66ad0
I20260926 14:43:19.099325 1433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:19.114550 1433 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:19.115839 1433 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:19.121989 1433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb: Generated new TSK 0
I20260926 14:43:19.122772 1433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:43:19.152877 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38279
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:19.259824 1447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:19.260100 1447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:19.260159 1447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:19.264146 1447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:19.264205 1447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:19.264284 1447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:43:19.269493 1447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:19.270531 1447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:19.271536 1447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:19.280830 1452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:19.280829 1453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:19.281100 1455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:19.281266 1447 server_base.cc:1034] running on GCE node
I20260926 14:43:19.281611 1447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:19.282315 1447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:19.283474 1447 hybrid_clock.cc:648] HybridClock initialized: now 1790433799283458 us; error 32 us; skew 500 ppm
Sep 26 14:43:19 dist-test-slave-x4sn krb5kdc[1340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433799, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:19.286154 1447 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:43:19.286959 1447 webserver.cc:460] Webserver started at http://127.16.90.193:34991/ using document root <none> and password file <none>
I20260926 14:43:19.287473 1447 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:19.287530 1447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:19.287750 1447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:19.289870 1447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "a469a870d91c4579a0b10660cda6d804"
format_stamp: "Formatted at 2026-09-26 14:43:19 on dist-test-slave-x4sn"
server_key: "fcf24daa96ff9c71572a038637670769"
server_key_iv: "4a9b0c0d10cde1761a3ad7a1696bf90c"
server_key_version: "encryptionkey@0"
I20260926 14:43:19.290395 1447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "a469a870d91c4579a0b10660cda6d804"
format_stamp: "Formatted at 2026-09-26 14:43:19 on dist-test-slave-x4sn"
server_key: "fcf24daa96ff9c71572a038637670769"
server_key_iv: "4a9b0c0d10cde1761a3ad7a1696bf90c"
server_key_version: "encryptionkey@0"
I20260926 14:43:19.293960 1447 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:43:19.296257 1462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:19.297417 1447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:19.297508 1447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "a469a870d91c4579a0b10660cda6d804"
format_stamp: "Formatted at 2026-09-26 14:43:19 on dist-test-slave-x4sn"
server_key: "fcf24daa96ff9c71572a038637670769"
server_key_iv: "4a9b0c0d10cde1761a3ad7a1696bf90c"
server_key_version: "encryptionkey@0"
I20260926 14:43:19.297586 1447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:19.308110 1447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:19.311054 1447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:19.311209 1447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:19.311775 1447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:19.312570 1447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:19.312618 1447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:19.312660 1447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:19.312690 1447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:19.320695 1447 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:42659
I20260926 14:43:19.320730 1575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:42659 every 8 connection(s)
I20260926 14:43:19.321554 1447 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:43:19.329064 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 1447
I20260926 14:43:19.329206 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:43:19.329474 16747 external_mini_cluster.cc:1428] Setting key d6d86780bcd5b65b7d0029ac1d4d2d43
Sep 26 14:43:19 dist-test-slave-x4sn krb5kdc[1340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433799, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:19.334849 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.323360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:48223 (local address 127.16.90.254:38279)
0926 14:43:19.323515 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:19.323518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:19.324322 (+ 804us) server_negotiation.cc:407] Connection header received
0926 14:43:19.325634 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:19.325638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:19.325694 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:19.325773 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:19.326909 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.327438 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.328599 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.328879 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.328986 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.331514 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:19.331517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:19.331520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:19.331548 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:19.332897 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:19.333445 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:19.333447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:19.333448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:19.333515 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:19.333833 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:19.333835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:19.333836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:19.333991 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:19.334058 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:19.334602 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 14:43:19.334715 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 14:43:19.336057 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.323728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38279 (local address 127.16.90.193:48223)
0926 14:43:19.324177 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:19.324220 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:19.325428 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:19.325950 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:19.325957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:19.326292 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:19.326762 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.326775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.327555 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.327558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.328478 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.328488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.329042 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.329046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.329199 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.329824 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:19.329841 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:19.331351 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:19.333010 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:19.333015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:19.333030 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:19.333362 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:19.333586 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:19.333589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:19.333591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:19.333761 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:19.334075 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:19.334080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:19.334381 (+ 301us) client_negotiation.cc:769] Sending connection context
0926 14:43:19.334589 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 14:43:19.334936 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260926 14:43:19.337267 1576 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38279
I20260926 14:43:19.337483 1576 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:19.337942 1576 heartbeater.cc:511] Master 127.16.90.254:38279 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
I20260926 14:43:19.339395 1385 ts_manager.cc:194] Registered new tserver with Master: a469a870d91c4579a0b10660cda6d804 (127.16.90.193:42659)
I20260926 14:43:19.340664 1385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:48223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:43:19.354197 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38279
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:19.483587 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:19.483984 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:19.484050 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:19.490373 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:19.490461 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:19.490562 1581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:43:19.498893 1581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:19.500383 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:19.501801 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:19.512528 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:19.512578 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:19.512538 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:19.513427 1581 server_base.cc:1034] running on GCE node
I20260926 14:43:19.513800 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:19.514400 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:19.515563 1581 hybrid_clock.cc:648] HybridClock initialized: now 1790433799515532 us; error 50 us; skew 500 ppm
Sep 26 14:43:19 dist-test-slave-x4sn krb5kdc[1340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433799, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:19.518437 1581 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:43:19.519512 1581 webserver.cc:460] Webserver started at http://127.16.90.194:42685/ using document root <none> and password file <none>
I20260926 14:43:19.520227 1581 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:19.520318 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:19.520586 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:19.523345 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "4ac830069df544df9fb13f9cfa67b478"
format_stamp: "Formatted at 2026-09-26 14:43:19 on dist-test-slave-x4sn"
server_key: "64d2a50ec92e3b2da33d9a7d3e042102"
server_key_iv: "6a5a591215a527091ffa52c32f873f8c"
server_key_version: "encryptionkey@0"
I20260926 14:43:19.524077 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "4ac830069df544df9fb13f9cfa67b478"
format_stamp: "Formatted at 2026-09-26 14:43:19 on dist-test-slave-x4sn"
server_key: "64d2a50ec92e3b2da33d9a7d3e042102"
server_key_iv: "6a5a591215a527091ffa52c32f873f8c"
server_key_version: "encryptionkey@0"
I20260926 14:43:19.528913 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 14:43:19.531987 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:19.533459 1581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:43:19.533591 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "4ac830069df544df9fb13f9cfa67b478"
format_stamp: "Formatted at 2026-09-26 14:43:19 on dist-test-slave-x4sn"
server_key: "64d2a50ec92e3b2da33d9a7d3e042102"
server_key_iv: "6a5a591215a527091ffa52c32f873f8c"
server_key_version: "encryptionkey@0"
I20260926 14:43:19.533712 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:19.565420 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:19.569446 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:19.569664 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:19.570417 1581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:19.571483 1581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:19.571556 1581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:19.571610 1581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:19.571640 1581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:19.582422 1581 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:44615
I20260926 14:43:19.582585 1711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:44615 every 8 connection(s)
I20260926 14:43:19.583611 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:43:19.590443 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 1581
I20260926 14:43:19.590579 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:43:19.590806 16747 external_mini_cluster.cc:1428] Setting key 4ef88f24e30411078917b057142e0b28
Sep 26 14:43:19 dist-test-slave-x4sn krb5kdc[1340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433799, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:19.599539 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.585622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:41709 (local address 127.16.90.254:38279)
0926 14:43:19.585774 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:19.585779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:19.586606 (+ 827us) server_negotiation.cc:407] Connection header received
0926 14:43:19.588008 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:19.588013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:19.588073 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:19.588158 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:19.589363 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.590244 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.592025 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.592363 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.592436 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.595542 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:19.595546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:19.595549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:19.595575 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:19.597221 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:19.597850 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:19.597854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:19.597855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:19.597940 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:19.598329 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:19.598332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:19.598333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:19.598557 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:19.598657 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:19.599257 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 14:43:19.599374 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
I20260926 14:43:19.600625 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.585993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38279 (local address 127.16.90.194:41709)
0926 14:43:19.586445 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:19.586507 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:19.587779 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:19.588308 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:19.588315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:19.588675 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:19.589228 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.589246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.590394 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.590398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.591880 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.591891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.592645 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.592649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.592853 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.593484 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:19.593505 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:19.595363 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:19.597357 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:19.597363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:19.597380 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:19.597752 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:19.598040 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:19.598044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:19.598046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:19.598234 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:19.598659 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:19.598666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:19.598984 (+ 318us) client_negotiation.cc:769] Sending connection context
0926 14:43:19.599191 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 14:43:19.599539 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
I20260926 14:43:19.601971 1712 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38279
I20260926 14:43:19.602221 1712 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
I20260926 14:43:19.602790 1712 heartbeater.cc:511] Master 127.16.90.254:38279 requested a full tablet report, sending...
I20260926 14:43:19.603942 1385 ts_manager.cc:194] Registered new tserver with Master: 4ac830069df544df9fb13f9cfa67b478 (127.16.90.194:44615)
I20260926 14:43:19.604552 1385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:41709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:43:19.616554 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38279
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:19.722210 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:19.722445 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:19.722483 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:19.726343 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:19.726392 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:19.726446 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:43:19.731465 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:38279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:19.732431 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:19.733351 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:19.742308 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:19.742343 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:19.742455 1717 server_base.cc:1034] running on GCE node
W20260926 14:43:19.742358 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:19.742866 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:19.743474 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:19.744616 1717 hybrid_clock.cc:648] HybridClock initialized: now 1790433799744598 us; error 53 us; skew 500 ppm
Sep 26 14:43:19 dist-test-slave-x4sn krb5kdc[1340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433799, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:19.747349 1717 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:43:19.748294 1717 webserver.cc:460] Webserver started at http://127.16.90.195:37867/ using document root <none> and password file <none>
I20260926 14:43:19.748936 1717 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:19.749011 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:19.749207 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:19.751262 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "4ca3301b6692406ab3095446fe9fd3dd"
format_stamp: "Formatted at 2026-09-26 14:43:19 on dist-test-slave-x4sn"
server_key: "42201d2d3b843eda3bb350786abf9cbd"
server_key_iv: "7718cdf1abd8a7a52440dd2569510d81"
server_key_version: "encryptionkey@0"
I20260926 14:43:19.751845 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "4ca3301b6692406ab3095446fe9fd3dd"
format_stamp: "Formatted at 2026-09-26 14:43:19 on dist-test-slave-x4sn"
server_key: "42201d2d3b843eda3bb350786abf9cbd"
server_key_iv: "7718cdf1abd8a7a52440dd2569510d81"
server_key_version: "encryptionkey@0"
I20260926 14:43:19.755525 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:19.757869 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:19.759042 1717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:19.759150 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "4ca3301b6692406ab3095446fe9fd3dd"
format_stamp: "Formatted at 2026-09-26 14:43:19 on dist-test-slave-x4sn"
server_key: "42201d2d3b843eda3bb350786abf9cbd"
server_key_iv: "7718cdf1abd8a7a52440dd2569510d81"
server_key_version: "encryptionkey@0"
I20260926 14:43:19.759243 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:19.806749 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:19.810029 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:19.810213 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:19.810825 1717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:19.811661 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:19.811745 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:19.811798 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:19.811821 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:19.820086 1717 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:45097
I20260926 14:43:19.820119 1845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:45097 every 8 connection(s)
I20260926 14:43:19.820930 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:43:19.822479 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 1717
I20260926 14:43:19.822604 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:43:19.822870 16747 external_mini_cluster.cc:1428] Setting key 680a370711ae14f011997a524095b697
Sep 26 14:43:19 dist-test-slave-x4sn krb5kdc[1340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433799, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:19.832937 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.822585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:54223 (local address 127.16.90.254:38279)
0926 14:43:19.822716 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:19.822720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:19.823610 (+ 890us) server_negotiation.cc:407] Connection header received
0926 14:43:19.824923 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:19.824925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:19.824969 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:19.825031 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:19.826092 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.826667 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.827755 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.827959 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.828028 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.830197 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:19.830199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:19.830201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:19.830225 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:19.831302 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:19.831759 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:19.831761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:19.831763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:19.831833 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:19.832090 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:19.832092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:19.832093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:19.832257 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:19.832324 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:19.832736 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 14:43:19.832825 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:43:19.833698 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.823002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:38279 (local address 127.16.90.195:54223)
0926 14:43:19.823454 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:19.823515 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:19.824756 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:19.825179 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:19.825186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:19.825532 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:19.825987 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.825999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.826751 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.826753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.827615 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.827623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.828067 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.828069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.828205 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.828712 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:19.828730 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:19.830052 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:19.831396 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:19.831400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:19.831411 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:19.831656 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:19.831895 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:19.831897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:19.831898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:19.832023 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:19.832331 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:19.832335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:19.832558 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 14:43:19.832699 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 14:43:19.832929 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":98,"threads_started":1}
I20260926 14:43:19.834684 1846 heartbeater.cc:347] Connected to a master server at 127.16.90.254:38279
I20260926 14:43:19.834889 1846 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:19.835340 1846 heartbeater.cc:511] Master 127.16.90.254:38279 requested a full tablet report, sending...
I20260926 14:43:19.836272 1385 ts_manager.cc:194] Registered new tserver with Master: 4ca3301b6692406ab3095446fe9fd3dd (127.16.90.195:45097)
I20260926 14:43:19.836797 1385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:54223
I20260926 14:43:19.847867 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:43:19.856288 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.849701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.16.90.254:38279)
0926 14:43:19.849813 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:19.849817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:19.849983 (+ 166us) server_negotiation.cc:407] Connection header received
0926 14:43:19.850095 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:19.850098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:19.850141 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:19.850203 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:19.850862 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.851365 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.852243 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.852452 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.852535 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.853744 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:19.853748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:19.853750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:19.853774 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:19.854790 (+ 1016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:19.855148 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:19.855151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:19.855153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:19.855224 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:19.855448 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:19.855451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:19.855453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:19.855702 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:19.855816 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:19.856036 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 14:43:19.856141 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:43:19.858727 1385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:43:19.861630 1385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:43:19.875131 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.869319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:42659 (local address 127.0.0.1:58618)
0926 14:43:19.869653 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:19.869710 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:19.869823 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:19.870885 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:19.870891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:19.870912 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:19.871029 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.871038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.872413 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.872416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.873733 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.873744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.874614 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.874617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.874737 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.874776 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 14:43:19.874874 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 14:43:19.874992 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":55,"thread_start_us":80,"threads_started":1}
I20260926 14:43:19.875140 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.869797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.16.90.193:42659)
0926 14:43:19.870412 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:19.870420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:19.870443 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:43:19.870514 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:19.870519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:19.870747 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:19.870886 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:19.871179 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.872306 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.873867 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.874508 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.874603 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.874746 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:19.874912 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 14:43:19.875004 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":81,"threads_started":1}
I20260926 14:43:19.875306 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.869422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:44615 (local address 127.0.0.1:47962)
0926 14:43:19.870284 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:19.870297 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:19.870378 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:19.871053 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:19.871056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:19.871074 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:19.871175 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.871182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.872260 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.872262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.873755 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.873762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.874933 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.874937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.875023 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.875045 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:43:19.875108 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:43:19.875176 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":92,"thread_start_us":96,"threads_started":1}
I20260926 14:43:19.875788 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.870008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47962 (local address 127.16.90.194:44615)
0926 14:43:19.870683 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:19.870695 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:19.870718 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:43:19.870787 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:19.870791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:19.870929 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:19.871062 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:19.871267 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.872144 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.873885 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.874634 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.874738 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.874868 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:19.875524 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 14:43:19.875633 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":100,"threads_started":1}
I20260926 14:43:19.875928 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.870057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:45097 (local address 127.0.0.1:52774)
0926 14:43:19.871102 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:19.871114 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:19.871197 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:19.871695 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:19.871698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:19.871717 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:19.871794 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.871800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.872969 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.872972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.873995 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.874005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.875177 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.875180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.875633 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.875656 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:43:19.875750 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 14:43:19.875814 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":60,"thread_start_us":105,"threads_started":1}
I20260926 14:43:19.876039 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.870317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.16.90.195:45097)
0926 14:43:19.870692 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:19.870702 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:19.871285 (+ 583us) server_negotiation.cc:407] Connection header received
0926 14:43:19.871359 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:19.871364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:19.871534 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:19.871702 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:19.871922 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.872831 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.874121 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.874898 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.874991 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.875122 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:19.875811 (+ 689us) server_negotiation.cc:299] Negotiation successful
0926 14:43:19.875907 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":88,"threads_started":1}
I20260926 14:43:19.878474 1510 tablet_service.cc:1467] Processing CreateTablet for tablet 70745d9960d3427f86c809d0acf5f4c2 (DEFAULT_TABLE table=test-table [id=dc748ad1056b42f4945ea97945b000f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:19.878474 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 70745d9960d3427f86c809d0acf5f4c2 (DEFAULT_TABLE table=test-table [id=dc748ad1056b42f4945ea97945b000f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:19.879566 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70745d9960d3427f86c809d0acf5f4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:19.879566 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70745d9960d3427f86c809d0acf5f4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:19.881402 1780 tablet_service.cc:1467] Processing CreateTablet for tablet 70745d9960d3427f86c809d0acf5f4c2 (DEFAULT_TABLE table=test-table [id=dc748ad1056b42f4945ea97945b000f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:19.882215 1780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70745d9960d3427f86c809d0acf5f4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:19.886654 1863 tablet_bootstrap.cc:492] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804: Bootstrap starting.
I20260926 14:43:19.887876 1865 tablet_bootstrap.cc:492] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd: Bootstrap starting.
I20260926 14:43:19.887939 1864 tablet_bootstrap.cc:492] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478: Bootstrap starting.
I20260926 14:43:19.888827 1863 tablet_bootstrap.cc:654] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:19.889498 1863 log.cc:824] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:19.890033 1865 tablet_bootstrap.cc:654] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:19.890064 1864 tablet_bootstrap.cc:654] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:19.891276 1865 log.cc:824] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:19.891281 1864 log.cc:824] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:19.891449 1863 tablet_bootstrap.cc:492] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804: No bootstrap required, opened a new log
I20260926 14:43:19.891634 1863 ts_tablet_manager.cc:1397] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:43:19.893097 1864 tablet_bootstrap.cc:492] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478: No bootstrap required, opened a new log
I20260926 14:43:19.893115 1865 tablet_bootstrap.cc:492] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd: No bootstrap required, opened a new log
I20260926 14:43:19.893278 1864 ts_tablet_manager.cc:1397] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:43:19.893278 1865 ts_tablet_manager.cc:1397] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:43:19.894920 1863 raft_consensus.cc:348] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a469a870d91c4579a0b10660cda6d804" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42659 } } peers { permanent_uuid: "4ac830069df544df9fb13f9cfa67b478" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 44615 } } peers { permanent_uuid: "4ca3301b6692406ab3095446fe9fd3dd" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 45097 } }
I20260926 14:43:19.895118 1863 raft_consensus.cc:374] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:19.895186 1863 raft_consensus.cc:729] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a469a870d91c4579a0b10660cda6d804, State: Initialized, Role: FOLLOWER
I20260926 14:43:19.895593 1863 consensus_queue.cc:260] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a469a870d91c4579a0b10660cda6d804" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42659 } } peers { permanent_uuid: "4ac830069df544df9fb13f9cfa67b478" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 44615 } } peers { permanent_uuid: "4ca3301b6692406ab3095446fe9fd3dd" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 45097 } }
I20260926 14:43:19.896195 1576 heartbeater.cc:503] Master 127.16.90.254:38279 was elected leader, sending a full tablet report...
I20260926 14:43:19.896869 1865 raft_consensus.cc:348] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a469a870d91c4579a0b10660cda6d804" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42659 } } peers { permanent_uuid: "4ac830069df544df9fb13f9cfa67b478" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 44615 } } peers { permanent_uuid: "4ca3301b6692406ab3095446fe9fd3dd" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 45097 } }
I20260926 14:43:19.896869 1864 raft_consensus.cc:348] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a469a870d91c4579a0b10660cda6d804" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42659 } } peers { permanent_uuid: "4ac830069df544df9fb13f9cfa67b478" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 44615 } } peers { permanent_uuid: "4ca3301b6692406ab3095446fe9fd3dd" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 45097 } }
I20260926 14:43:19.897118 1864 raft_consensus.cc:374] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:19.897147 1865 raft_consensus.cc:374] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:19.897200 1864 raft_consensus.cc:729] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac830069df544df9fb13f9cfa67b478, State: Initialized, Role: FOLLOWER
I20260926 14:43:19.897208 1865 raft_consensus.cc:729] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ca3301b6692406ab3095446fe9fd3dd, State: Initialized, Role: FOLLOWER
I20260926 14:43:19.897375 1863 ts_tablet_manager.cc:1428] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:43:19.897590 1865 consensus_queue.cc:260] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a469a870d91c4579a0b10660cda6d804" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42659 } } peers { permanent_uuid: "4ac830069df544df9fb13f9cfa67b478" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 44615 } } peers { permanent_uuid: "4ca3301b6692406ab3095446fe9fd3dd" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 45097 } }
I20260926 14:43:19.897621 1864 consensus_queue.cc:260] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a469a870d91c4579a0b10660cda6d804" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42659 } } peers { permanent_uuid: "4ac830069df544df9fb13f9cfa67b478" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 44615 } } peers { permanent_uuid: "4ca3301b6692406ab3095446fe9fd3dd" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 45097 } }
I20260926 14:43:19.898306 1846 heartbeater.cc:503] Master 127.16.90.254:38279 was elected leader, sending a full tablet report...
I20260926 14:43:19.898548 1865 ts_tablet_manager.cc:1428] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260926 14:43:19.898869 1864 ts_tablet_manager.cc:1428] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:43:19.899147 1712 heartbeater.cc:503] Master 127.16.90.254:38279 was elected leader, sending a full tablet report...
I20260926 14:43:19.989965 1871 raft_consensus.cc:482] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:43:19.990177 1871 raft_consensus.cc:504] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a469a870d91c4579a0b10660cda6d804" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42659 } } peers { permanent_uuid: "4ac830069df544df9fb13f9cfa67b478" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 44615 } } peers { permanent_uuid: "4ca3301b6692406ab3095446fe9fd3dd" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 45097 } }
I20260926 14:43:19.991341 1871 leader_election.cc:288] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a469a870d91c4579a0b10660cda6d804 (127.16.90.193:42659), 4ca3301b6692406ab3095446fe9fd3dd (127.16.90.195:45097)
I20260926 14:43:19.994867 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.991728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:33217 (local address 127.16.90.193:42659)
0926 14:43:19.991848 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:19.991853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:19.991870 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:43:19.991926 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:19.991929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:19.991979 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:19.992057 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:19.992360 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.992852 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.993926 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.994343 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.994448 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.994502 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:19.994687 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 14:43:19.994763 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:43:19.994867 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.991551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:42659 (local address 127.16.90.194:33217)
0926 14:43:19.991705 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:19.991721 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:19.991814 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:19.992099 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:19.992102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:19.992133 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:19.992248 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.992254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.992958 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.992961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.993801 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.993810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.994452 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.994456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.994563 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.994594 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 14:43:19.994658 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:43:19.994733 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":39}
I20260926 14:43:19.995205 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.991960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:56445 (local address 127.16.90.195:45097)
0926 14:43:19.992104 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:19.992109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:19.992288 (+ 179us) server_negotiation.cc:407] Connection header received
0926 14:43:19.992389 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:19.992392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:19.992435 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:19.992497 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:19.992760 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.993234 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.994277 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.994672 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.994773 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.994821 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:19.995012 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 14:43:19.995086 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:43:19.995512 1872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:19.991847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:45097 (local address 127.16.90.194:56445)
0926 14:43:19.992204 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:19.992215 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:19.992304 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:19.992514 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:19.992517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:19.992532 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:19.992656 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.992662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.993335 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.993338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.994155 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:19.994171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:19.994777 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:19.994780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:19.994873 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:19.994894 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:43:19.995343 (+ 449us) client_negotiation.cc:240] Negotiation successful
0926 14:43:19.995408 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":21,"thread_start_us":121,"threads_started":1}
I20260926 14:43:19.995510 1530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70745d9960d3427f86c809d0acf5f4c2" candidate_uuid: "4ac830069df544df9fb13f9cfa67b478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a469a870d91c4579a0b10660cda6d804" is_pre_election: true
I20260926 14:43:19.995857 1530 raft_consensus.cc:2454] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ac830069df544df9fb13f9cfa67b478 in term 0.
I20260926 14:43:19.996084 1800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70745d9960d3427f86c809d0acf5f4c2" candidate_uuid: "4ac830069df544df9fb13f9cfa67b478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ca3301b6692406ab3095446fe9fd3dd" is_pre_election: true
I20260926 14:43:19.996359 1800 raft_consensus.cc:2454] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ac830069df544df9fb13f9cfa67b478 in term 0.
I20260926 14:43:19.996338 1599 leader_election.cc:302] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac830069df544df9fb13f9cfa67b478, a469a870d91c4579a0b10660cda6d804; no voters:
I20260926 14:43:19.996626 1871 raft_consensus.cc:2781] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:43:19.996729 1871 raft_consensus.cc:482] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:43:19.996836 1871 raft_consensus.cc:3037] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:19.997854 1871 raft_consensus.cc:504] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a469a870d91c4579a0b10660cda6d804" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42659 } } peers { permanent_uuid: "4ac830069df544df9fb13f9cfa67b478" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 44615 } } peers { permanent_uuid: "4ca3301b6692406ab3095446fe9fd3dd" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 45097 } }
I20260926 14:43:19.998174 1871 leader_election.cc:288] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [CANDIDATE]: Term 1 election: Requested vote from peers a469a870d91c4579a0b10660cda6d804 (127.16.90.193:42659), 4ca3301b6692406ab3095446fe9fd3dd (127.16.90.195:45097)
I20260926 14:43:19.998523 1530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70745d9960d3427f86c809d0acf5f4c2" candidate_uuid: "4ac830069df544df9fb13f9cfa67b478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a469a870d91c4579a0b10660cda6d804"
I20260926 14:43:19.998550 1800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70745d9960d3427f86c809d0acf5f4c2" candidate_uuid: "4ac830069df544df9fb13f9cfa67b478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ca3301b6692406ab3095446fe9fd3dd"
I20260926 14:43:19.998656 1800 raft_consensus.cc:3037] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:19.998667 1530 raft_consensus.cc:3037] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:19.999892 1530 raft_consensus.cc:2454] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ac830069df544df9fb13f9cfa67b478 in term 1.
I20260926 14:43:19.999892 1800 raft_consensus.cc:2454] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ac830069df544df9fb13f9cfa67b478 in term 1.
I20260926 14:43:20.000196 1599 leader_election.cc:302] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac830069df544df9fb13f9cfa67b478, a469a870d91c4579a0b10660cda6d804; no voters:
I20260926 14:43:20.000344 1871 raft_consensus.cc:2781] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:20.000554 1871 raft_consensus.cc:686] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 1 LEADER]: Becoming Leader. State: Replica: 4ac830069df544df9fb13f9cfa67b478, State: Running, Role: LEADER
I20260926 14:43:20.000885 1871 consensus_queue.cc:237] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a469a870d91c4579a0b10660cda6d804" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42659 } } peers { permanent_uuid: "4ac830069df544df9fb13f9cfa67b478" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 44615 } } peers { permanent_uuid: "4ca3301b6692406ab3095446fe9fd3dd" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 45097 } }
I20260926 14:43:20.004060 1385 catalog_manager.cc:5495] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ac830069df544df9fb13f9cfa67b478 (127.16.90.194). New cstate: current_term: 1 leader_uuid: "4ac830069df544df9fb13f9cfa67b478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a469a870d91c4579a0b10660cda6d804" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ac830069df544df9fb13f9cfa67b478" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 44615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ca3301b6692406ab3095446fe9fd3dd" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 45097 } health_report { overall_health: UNKNOWN } } }
I20260926 14:43:20.026180 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.022863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47976 (local address 127.16.90.194:44615)
0926 14:43:20.022984 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:20.022988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:20.023002 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:43:20.023079 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:20.023081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:20.023134 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:20.023224 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:20.023444 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.023988 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.024823 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.025031 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.025114 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.025396 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:20.025804 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:20.025908 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:20.025977 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 14:43:20.026033 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:43:20.031770 1530 raft_consensus.cc:1261] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804 [term 1 FOLLOWER]: Refusing update from remote peer 4ac830069df544df9fb13f9cfa67b478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:20.031770 1800 raft_consensus.cc:1261] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd [term 1 FOLLOWER]: Refusing update from remote peer 4ac830069df544df9fb13f9cfa67b478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:20.032446 1871 consensus_queue.cc:1035] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a469a870d91c4579a0b10660cda6d804" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:20.032629 1873 consensus_queue.cc:1035] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ca3301b6692406ab3095446fe9fd3dd" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 45097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:20.039947 1877 mvcc.cc:205] Tried to move back new op lower bound from 7333616844923027456 to 7333616844806324224. Current Snapshot: MvccSnapshot[applied={T|T < 7333616844923027456}]
I20260926 14:43:20.049605 1382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:43:20.049780 1382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:43:20.051993 1382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa155a73e3ad48179dbd49fb30d58bcb: Sending DeleteTablet for 3 replicas of tablet 70745d9960d3427f86c809d0acf5f4c2
I20260926 14:43:20.052713 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 70745d9960d3427f86c809d0acf5f4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:20 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:47962
I20260926 14:43:20.052767 1780 tablet_service.cc:1514] Processing DeleteTablet for tablet 70745d9960d3427f86c809d0acf5f4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:20 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:52774
I20260926 14:43:20.053216 1886 tablet_replica.cc:330] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd: stopping tablet replica
I20260926 14:43:20.053220 1510 tablet_service.cc:1514] Processing DeleteTablet for tablet 70745d9960d3427f86c809d0acf5f4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:20 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:58618
I20260926 14:43:20.053553 1886 raft_consensus.cc:2229] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:43:20.053717 1888 tablet_replica.cc:330] T 70745d9960d3427f86c809d0acf5f4c2 P a469a870d91c4579a0b10660cda6d804: stopping tablet replica
I20260926 14:43:20.053900 1886 raft_consensus.cc:2258] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:43:20.054253 1887 tablet_replica.cc:330] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478: stopping tablet replica
I20260926 14:43:20.054375 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 1447
I20260926 14:43:20.054904 1887 raft_consensus.cc:2229] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:43:20.055289 1887 raft_consensus.cc:2258] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:43:20.055909 1886 ts_tablet_manager.cc:1905] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:43:20.058570 1886 ts_tablet_manager.cc:1918] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:43:20.058755 1886 log.cc:1196] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/70745d9960d3427f86c809d0acf5f4c2
I20260926 14:43:20.059144 1886 ts_tablet_manager.cc:1939] T 70745d9960d3427f86c809d0acf5f4c2 P 4ca3301b6692406ab3095446fe9fd3dd: Deleting consensus metadata
I20260926 14:43:20.059126 1887 ts_tablet_manager.cc:1905] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:43:20.060560 1371 catalog_manager.cc:4841] TS 4ca3301b6692406ab3095446fe9fd3dd (127.16.90.195:45097): tablet 70745d9960d3427f86c809d0acf5f4c2 (table test-table [id=dc748ad1056b42f4945ea97945b000f2]) successfully deleted
I20260926 14:43:20.061568 1887 ts_tablet_manager.cc:1918] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:43:20.061661 1887 log.cc:1196] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/70745d9960d3427f86c809d0acf5f4c2
I20260926 14:43:20.061959 1887 ts_tablet_manager.cc:1939] T 70745d9960d3427f86c809d0acf5f4c2 P 4ac830069df544df9fb13f9cfa67b478: Deleting consensus metadata
I20260926 14:43:20.063047 1372 catalog_manager.cc:4841] TS 4ac830069df544df9fb13f9cfa67b478 (127.16.90.194:44615): tablet 70745d9960d3427f86c809d0acf5f4c2 (table test-table [id=dc748ad1056b42f4945ea97945b000f2]) successfully deleted
W20260926 14:43:20.063512 1371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:42659 (error 108)
I20260926 14:43:20.064378 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 1581
I20260926 14:43:20.064522 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.064251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:42659 (local address 127.0.0.1:58622)
0926 14:43:20.064371 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:20.064437 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 14:43:20.064965 1371 catalog_manager.cc:4584] TS a469a870d91c4579a0b10660cda6d804 (127.16.90.193:42659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70745d9960d3427f86c809d0acf5f4c2: Network error: Client connection negotiation failed: client connection to 127.16.90.193:42659: connect: Connection refused (error 111)
I20260926 14:43:20.069588 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 1717
I20260926 14:43:20.074708 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 1353
2026-09-26T14:43:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2249 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T14:43:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:20Z Disabled control of system clock
I20260926 14:43:20.102953 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:39443
--webserver_interface=127.16.90.254
--webserver_port=0
--builtin_ntp_servers=127.16.90.212:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:39443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:43:20.214210 1893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:20.214480 1893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:20.214516 1893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:20.218532 1893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:20.218593 1893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:20.218608 1893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:20.218626 1893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:20.218653 1893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:20.223415 1893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:39443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:39443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:20.224485 1893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:20.225833 1893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:20.234205 1899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:20.234218 1898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:20.234212 1901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:20.234472 1893 server_base.cc:1034] running on GCE node
I20260926 14:43:20.235085 1893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:20.236227 1893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:20.237371 1893 hybrid_clock.cc:648] HybridClock initialized: now 1790433800237348 us; error 41 us; skew 500 ppm
I20260926 14:43:20.239046 1893 webserver.cc:460] Webserver started at http://127.16.90.254:41317/ using document root <none> and password file <none>
I20260926 14:43:20.239590 1893 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:20.239656 1893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:20.239884 1893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:20.241873 1893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "8e10030946d1491cbfa691916486a2d8"
format_stamp: "Formatted at 2026-09-26 14:43:20 on dist-test-slave-x4sn"
server_key: "95cedc1b2f72c114c0dfdae4a9b138e3"
server_key_iv: "f3f120cf4ee727e4d063b5d86fed2e22"
server_key_version: "encryptionkey@0"
I20260926 14:43:20.242395 1893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "8e10030946d1491cbfa691916486a2d8"
format_stamp: "Formatted at 2026-09-26 14:43:20 on dist-test-slave-x4sn"
server_key: "95cedc1b2f72c114c0dfdae4a9b138e3"
server_key_iv: "f3f120cf4ee727e4d063b5d86fed2e22"
server_key_version: "encryptionkey@0"
I20260926 14:43:20.245884 1893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 14:43:20.248227 1907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:20.249400 1893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:20.249509 1893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "8e10030946d1491cbfa691916486a2d8"
format_stamp: "Formatted at 2026-09-26 14:43:20 on dist-test-slave-x4sn"
server_key: "95cedc1b2f72c114c0dfdae4a9b138e3"
server_key_iv: "f3f120cf4ee727e4d063b5d86fed2e22"
server_key_version: "encryptionkey@0"
I20260926 14:43:20.249599 1893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:20.263408 1893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:20.264146 1893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:20.264331 1893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:20.270957 1893 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:39443
I20260926 14:43:20.270974 1960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:39443 every 8 connection(s)
I20260926 14:43:20.271970 1893 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:20.274936 1961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:20.277972 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 1893
I20260926 14:43:20.278108 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:20.278407 16747 external_mini_cluster.cc:1428] Setting key bfe4f6310558eb3eeaf5f0ce839b12c9
I20260926 14:43:20.281121 1961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8: Bootstrap starting.
I20260926 14:43:20.283249 1961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:20.283999 1961 log.cc:824] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:20.285794 1961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8: No bootstrap required, opened a new log
I20260926 14:43:20.287572 1964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.279829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.16.90.254:39443)
0926 14:43:20.280480 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:20.280495 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:20.280531 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:43:20.281142 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:20.281166 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:20.281582 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:20.281901 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:20.282331 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.283277 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.284181 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.284789 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.284933 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.285688 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:20.285693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:20.285705 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:20.286097 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:20.286150 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:20.286178 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:20.286259 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:20.286382 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 14:43:20.286645 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":137,"threads_started":1}
I20260926 14:43:20.290176 1961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e10030946d1491cbfa691916486a2d8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39443 } }
I20260926 14:43:20.290444 1961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:20.290503 1961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e10030946d1491cbfa691916486a2d8, State: Initialized, Role: FOLLOWER
I20260926 14:43:20.290927 1961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e10030946d1491cbfa691916486a2d8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39443 } }
I20260926 14:43:20.291050 1961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:20.291115 1961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:20.291213 1961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:20.292382 1961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e10030946d1491cbfa691916486a2d8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39443 } }
I20260926 14:43:20.292754 1961 leader_election.cc:302] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e10030946d1491cbfa691916486a2d8; no voters:
I20260926 14:43:20.293103 1961 leader_election.cc:288] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:20.293215 1966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:20.293426 1966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 8e10030946d1491cbfa691916486a2d8, State: Running, Role: LEADER
I20260926 14:43:20.293748 1966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e10030946d1491cbfa691916486a2d8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39443 } }
I20260926 14:43:20.294391 1961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:20.295338 1967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e10030946d1491cbfa691916486a2d8. Latest consensus state: current_term: 1 leader_uuid: "8e10030946d1491cbfa691916486a2d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e10030946d1491cbfa691916486a2d8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39443 } } }
I20260926 14:43:20.295461 1967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:20.295828 1975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:20.296192 1968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e10030946d1491cbfa691916486a2d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e10030946d1491cbfa691916486a2d8" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39443 } } }
I20260926 14:43:20.296372 1968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:20.298843 1975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:20.304677 1975 catalog_manager.cc:1347] Generated new cluster ID: cc000124ea7a4b8d877c0f33595fbd1c
I20260926 14:43:20.304754 1975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:20.329102 1975 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:20.330385 1975 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:20.337622 1975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8: Generated new TSK 0
I20260926 14:43:20.338304 1975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:20.347039 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:39443
--builtin_ntp_servers=127.16.90.212:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:43:20.457753 1985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:20.458113 1985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:20.458180 1985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:20.462827 1985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:20.462889 1985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:20.462946 1985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:43:20.467738 1985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:39443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:20.468791 1985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:20.470136 1985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:20.479550 1991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:20.479598 1990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:20.479550 1993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:20.479750 1985 server_base.cc:1034] running on GCE node
I20260926 14:43:20.480257 1985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:20.480873 1985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:20.482038 1985 hybrid_clock.cc:648] HybridClock initialized: now 1790433800482011 us; error 39 us; skew 500 ppm
I20260926 14:43:20.483973 1985 webserver.cc:460] Webserver started at http://127.16.90.193:38857/ using document root <none> and password file <none>
I20260926 14:43:20.484526 1985 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:20.484596 1985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:20.484795 1985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:20.486866 1985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c"
format_stamp: "Formatted at 2026-09-26 14:43:20 on dist-test-slave-x4sn"
server_key: "2c736485d577b572b8cb1b6bc201026c"
server_key_iv: "0ca1b76582c2cada68c01c96e1db0c7b"
server_key_version: "encryptionkey@0"
I20260926 14:43:20.487417 1985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c"
format_stamp: "Formatted at 2026-09-26 14:43:20 on dist-test-slave-x4sn"
server_key: "2c736485d577b572b8cb1b6bc201026c"
server_key_iv: "0ca1b76582c2cada68c01c96e1db0c7b"
server_key_version: "encryptionkey@0"
I20260926 14:43:20.491159 1985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:43:20.493628 1999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:20.494889 1985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:20.495004 1985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c"
format_stamp: "Formatted at 2026-09-26 14:43:20 on dist-test-slave-x4sn"
server_key: "2c736485d577b572b8cb1b6bc201026c"
server_key_iv: "0ca1b76582c2cada68c01c96e1db0c7b"
server_key_version: "encryptionkey@0"
I20260926 14:43:20.495146 1985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:20.514219 1985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:20.514942 1985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:20.515125 1985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:20.515767 1985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:20.516737 1985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:20.516785 1985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:20.516837 1985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:20.516862 1985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:20.525051 1985 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:34997
I20260926 14:43:20.525097 2112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:34997 every 8 connection(s)
I20260926 14:43:20.525924 1985 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:43:20.533262 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 1985
I20260926 14:43:20.533365 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:43:20.533617 16747 external_mini_cluster.cc:1428] Setting key 06594eafff5d9f5892e13141e82b2846
I20260926 14:43:20.534422 1964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.527646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:34605 (local address 127.16.90.254:39443)
0926 14:43:20.527809 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:20.527813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:20.528867 (+ 1054us) server_negotiation.cc:407] Connection header received
0926 14:43:20.530176 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:20.530179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:20.530226 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:20.530286 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:20.531122 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.531650 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.532727 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.532942 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.533027 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.534013 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:20.534016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:20.534018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:20.534030 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:20.534049 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:20.534057 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:20.534117 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:20.534257 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 14:43:20.534316 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 14:43:20.535363 2115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.528070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:39443 (local address 127.16.90.193:34605)
0926 14:43:20.528723 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:20.528764 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:20.529977 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:20.530433 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:20.530439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:20.530523 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:20.531012 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:20.531024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.531768 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.531770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:20.532625 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:20.532634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.533052 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.533054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:20.533185 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.533768 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:43:20.533769 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:20.533808 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:43:20.533871 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:20.534132 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:20.534140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:20.534145 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:43:20.534241 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 14:43:20.534506 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
I20260926 14:43:20.535506 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:39443
--builtin_ntp_servers=127.16.90.212:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:43:20.536468 2113 heartbeater.cc:347] Connected to a master server at 127.16.90.254:39443
I20260926 14:43:20.536715 2113 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:20.537202 2113 heartbeater.cc:511] Master 127.16.90.254:39443 requested a full tablet report, sending...
I20260926 14:43:20.538744 1924 ts_manager.cc:194] Registered new tserver with Master: 18a1cb3db86b4bd4981a5a7b3095ff0c (127.16.90.193:34997)
I20260926 14:43:20.540302 1924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.90.193:34605
W20260926 14:43:20.641526 2116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:20.641777 2116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:20.641813 2116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:20.645882 2116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:20.645955 2116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:20.646011 2116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:43:20.650686 2116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:39443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:20.651698 2116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:20.653091 2116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:20.662021 2124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:20.662045 2122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:20.662041 2121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:20.662357 2116 server_base.cc:1034] running on GCE node
I20260926 14:43:20.662739 2116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:20.663323 2116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:20.664470 2116 hybrid_clock.cc:648] HybridClock initialized: now 1790433800664461 us; error 37 us; skew 500 ppm
I20260926 14:43:20.666118 2116 webserver.cc:460] Webserver started at http://127.16.90.194:42767/ using document root <none> and password file <none>
I20260926 14:43:20.666642 2116 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:20.666708 2116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:20.666888 2116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:20.668880 2116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "67e1405f49454a4488649893edda9e5f"
format_stamp: "Formatted at 2026-09-26 14:43:20 on dist-test-slave-x4sn"
server_key: "f0627eb0b4226f549de87d584342a429"
server_key_iv: "26d7e94a8610ee636489bc003ea8d9c7"
server_key_version: "encryptionkey@0"
I20260926 14:43:20.669389 2116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "67e1405f49454a4488649893edda9e5f"
format_stamp: "Formatted at 2026-09-26 14:43:20 on dist-test-slave-x4sn"
server_key: "f0627eb0b4226f549de87d584342a429"
server_key_iv: "26d7e94a8610ee636489bc003ea8d9c7"
server_key_version: "encryptionkey@0"
I20260926 14:43:20.672775 2116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:43:20.674939 2130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:20.676237 2116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:20.676337 2116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "67e1405f49454a4488649893edda9e5f"
format_stamp: "Formatted at 2026-09-26 14:43:20 on dist-test-slave-x4sn"
server_key: "f0627eb0b4226f549de87d584342a429"
server_key_iv: "26d7e94a8610ee636489bc003ea8d9c7"
server_key_version: "encryptionkey@0"
I20260926 14:43:20.676427 2116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:20.693372 2116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:20.693995 2116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:20.694164 2116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:20.694747 2116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:20.695724 2116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:20.695772 2116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:20.695822 2116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:20.695849 2116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:20.703733 2116 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:43169
I20260926 14:43:20.703763 2243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:43169 every 8 connection(s)
I20260926 14:43:20.704510 2116 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:43:20.711531 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 2116
I20260926 14:43:20.711684 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:43:20.711951 16747 external_mini_cluster.cc:1428] Setting key da48549a9e08457eb7c2577269688e03
I20260926 14:43:20.712987 1964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.706117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:39611 (local address 127.16.90.254:39443)
0926 14:43:20.706227 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:20.706230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:20.707177 (+ 947us) server_negotiation.cc:407] Connection header received
0926 14:43:20.708701 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:20.708703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:20.708750 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:20.708817 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:20.709637 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.710147 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.711205 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.711474 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.711578 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.712574 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:20.712577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:20.712579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:20.712591 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:20.712609 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:20.712617 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:20.712682 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:20.712821 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 14:43:20.712879 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:43:20.713928 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.706443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:39443 (local address 127.16.90.194:39611)
0926 14:43:20.707024 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:20.707062 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:20.708499 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:20.708960 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:20.708967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:20.709056 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:20.709519 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:20.709533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.710263 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.710266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:20.711115 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:20.711124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.711609 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.711613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:20.711824 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.712322 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:43:20.712324 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:20.712362 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:43:20.712423 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:20.712690 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:20.712699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:20.712704 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:43:20.712804 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 14:43:20.713055 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":104,"threads_started":1}
I20260926 14:43:20.714115 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:39443
--builtin_ntp_servers=127.16.90.212:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:43:20.716079 2244 heartbeater.cc:347] Connected to a master server at 127.16.90.254:39443
I20260926 14:43:20.716286 2244 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:20.716748 2244 heartbeater.cc:511] Master 127.16.90.254:39443 requested a full tablet report, sending...
I20260926 14:43:20.717705 1924 ts_manager.cc:194] Registered new tserver with Master: 67e1405f49454a4488649893edda9e5f (127.16.90.194:43169)
I20260926 14:43:20.718231 1924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.90.194:39611
W20260926 14:43:20.819411 2247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:20.819700 2247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:20.819766 2247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:20.823729 2247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:20.823786 2247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:20.823860 2247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:43:20.828544 2247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:39443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:20.829634 2247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:20.831022 2247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:20.840047 2253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:20.840040 2255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:20.840065 2252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:20.840392 2247 server_base.cc:1034] running on GCE node
I20260926 14:43:20.840799 2247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:20.841406 2247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:20.842555 2247 hybrid_clock.cc:648] HybridClock initialized: now 1790433800842526 us; error 48 us; skew 500 ppm
I20260926 14:43:20.844301 2247 webserver.cc:460] Webserver started at http://127.16.90.195:44869/ using document root <none> and password file <none>
I20260926 14:43:20.844847 2247 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:20.844911 2247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:20.845098 2247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:20.847106 2247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "13dc473b4dc242169d6b1cf6d8b673ec"
format_stamp: "Formatted at 2026-09-26 14:43:20 on dist-test-slave-x4sn"
server_key: "54ce8f469736e775ed047b945a257ebd"
server_key_iv: "d214bc9bd2b77d20cde260ba718581be"
server_key_version: "encryptionkey@0"
I20260926 14:43:20.847638 2247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "13dc473b4dc242169d6b1cf6d8b673ec"
format_stamp: "Formatted at 2026-09-26 14:43:20 on dist-test-slave-x4sn"
server_key: "54ce8f469736e775ed047b945a257ebd"
server_key_iv: "d214bc9bd2b77d20cde260ba718581be"
server_key_version: "encryptionkey@0"
I20260926 14:43:20.851183 2247 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:20.853621 2261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:20.854796 2247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:20.854904 2247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "13dc473b4dc242169d6b1cf6d8b673ec"
format_stamp: "Formatted at 2026-09-26 14:43:20 on dist-test-slave-x4sn"
server_key: "54ce8f469736e775ed047b945a257ebd"
server_key_iv: "d214bc9bd2b77d20cde260ba718581be"
server_key_version: "encryptionkey@0"
I20260926 14:43:20.855001 2247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:20.873204 2247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:20.873905 2247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:20.874076 2247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:20.874655 2247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:20.875576 2247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:20.875648 2247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:20.875717 2247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:20.875746 2247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:20.883965 2247 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:33949
I20260926 14:43:20.884001 2376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:33949 every 8 connection(s)
I20260926 14:43:20.884840 2247 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:43:20.889334 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 2247
I20260926 14:43:20.889424 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:43:20.889676 16747 external_mini_cluster.cc:1428] Setting key 7ee4a56cbd1ccd5fc72e51be700f5497
I20260926 14:43:20.893342 1964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.886487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:58283 (local address 127.16.90.254:39443)
0926 14:43:20.886616 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:20.886620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:20.887634 (+ 1014us) server_negotiation.cc:407] Connection header received
0926 14:43:20.888999 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:20.889002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:20.889057 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:20.889139 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:20.890057 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.890580 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.891686 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.891909 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.891994 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.892915 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:20.892918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:20.892921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:20.892934 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:20.892951 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:20.892959 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:20.893015 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:20.893148 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 14:43:20.893205 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:43:20.894147 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.886877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:39443 (local address 127.16.90.195:58283)
0926 14:43:20.887482 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:20.887521 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:20.888792 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:20.889313 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:20.889319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:20.889406 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:20.889924 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:20.889937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.890692 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.890695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:20.891545 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:20.891554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.892037 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.892040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:20.892194 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.892683 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:43:20.892684 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:20.892723 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:43:20.892783 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:20.893024 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:20.893032 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:20.893038 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:43:20.893142 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 14:43:20.893431 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":103,"threads_started":1}
I20260926 14:43:20.895131 2377 heartbeater.cc:347] Connected to a master server at 127.16.90.254:39443
I20260926 14:43:20.895340 2377 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:20.895876 2377 heartbeater.cc:511] Master 127.16.90.254:39443 requested a full tablet report, sending...
I20260926 14:43:20.896772 1924 ts_manager.cc:194] Registered new tserver with Master: 13dc473b4dc242169d6b1cf6d8b673ec (127.16.90.195:33949)
I20260926 14:43:20.897308 1924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.90.195:58283
I20260926 14:43:20.902973 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:43:20.908221 1964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.904784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49622 (local address 127.16.90.254:39443)
0926 14:43:20.904906 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:20.904910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:20.905059 (+ 149us) server_negotiation.cc:407] Connection header received
0926 14:43:20.905176 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:20.905179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:20.905222 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:20.905283 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:20.905534 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.906027 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.906824 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.907041 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.907123 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.907846 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:20.907850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:20.907852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:20.907864 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:20.907885 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:20.907892 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:20.907948 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:20.908032 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 14:43:20.908084 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:43:20.910615 1924 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:43:20.913646 1924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:43:20.926798 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.921377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:34997 (local address 127.0.0.1:50994)
0926 14:43:20.922096 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:20.922108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:20.922188 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:20.922864 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:20.922867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:20.922885 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:20.922979 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:20.922985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.924455 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.924457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:20.925321 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:20.925330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.926262 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.926266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:20.926369 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.926406 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 14:43:20.926508 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:43:20.926647 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260926 14:43:20.926991 2390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.921877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:43169 (local address 127.0.0.1:41244)
0926 14:43:20.922212 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:20.922223 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:20.922290 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:20.923130 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:20.923132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:20.923149 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:20.923233 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:20.923239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.924185 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.924188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:20.925461 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:20.925471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.926520 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.926524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:20.926682 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.926706 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:43:20.926757 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 14:43:20.926810 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":22,"thread_start_us":88,"threads_started":1}
I20260926 14:43:20.927313 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.921559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.16.90.193:34997)
0926 14:43:20.921941 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:20.921954 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:20.922279 (+ 325us) server_negotiation.cc:407] Connection header received
0926 14:43:20.922360 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:20.922365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:20.922560 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:20.922718 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:20.923090 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.924309 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.925448 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.926134 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.926241 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.926438 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:20.926647 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 14:43:20.926747 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
I20260926 14:43:20.927340 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.922278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.16.90.194:43169)
0926 14:43:20.922736 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:20.922748 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:20.922768 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:43:20.922834 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:20.922838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:20.922992 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:20.923153 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:20.923316 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.924086 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.925602 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.926386 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.926859 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.927040 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:20.927090 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 14:43:20.927188 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":116,"threads_started":1}
I20260926 14:43:20.927557 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.921159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33949 (local address 127.0.0.1:51254)
0926 14:43:20.921829 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:20.921879 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:20.921987 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:20.922638 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:20.922643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:20.922667 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:20.922806 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:20.922814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.924412 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.924415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:20.925494 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:20.925501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.926515 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.926519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:20.927298 (+ 779us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.927319 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:43:20.927382 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:43:20.927448 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":667,"thread_start_us":82,"threads_started":1}
I20260926 14:43:20.928068 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:20.921297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.16.90.195:33949)
0926 14:43:20.922125 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:20.922136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:20.922160 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:43:20.922237 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:20.922243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:20.922454 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:20.922946 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:20.923090 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.924306 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.925606 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:20.926386 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:20.926486 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:20.926662 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:20.927823 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0926 14:43:20.927918 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":156,"threads_started":1}
I20260926 14:43:20.929991 2047 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f6f8b6f604bcfa36ae63a27f9d88d (DEFAULT_TABLE table=test-table [id=f21100d8f1d0433fbc301877b5ce7482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:20.930125 2178 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f6f8b6f604bcfa36ae63a27f9d88d (DEFAULT_TABLE table=test-table [id=f21100d8f1d0433fbc301877b5ce7482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:20.930174 2309 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f6f8b6f604bcfa36ae63a27f9d88d (DEFAULT_TABLE table=test-table [id=f21100d8f1d0433fbc301877b5ce7482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:20.930884 2047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f6f8b6f604bcfa36ae63a27f9d88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:20.931088 2309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f6f8b6f604bcfa36ae63a27f9d88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:20.931088 2178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f6f8b6f604bcfa36ae63a27f9d88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:20.937880 2394 tablet_bootstrap.cc:492] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c: Bootstrap starting.
I20260926 14:43:20.938552 2396 tablet_bootstrap.cc:492] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f: Bootstrap starting.
I20260926 14:43:20.938552 2395 tablet_bootstrap.cc:492] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec: Bootstrap starting.
I20260926 14:43:20.939956 2394 tablet_bootstrap.cc:654] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:20.940651 2394 log.cc:824] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:20.940681 2396 tablet_bootstrap.cc:654] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:20.940672 2395 tablet_bootstrap.cc:654] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:20.941459 2395 log.cc:824] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:20.941459 2396 log.cc:824] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:20.942575 2394 tablet_bootstrap.cc:492] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c: No bootstrap required, opened a new log
I20260926 14:43:20.942777 2394 ts_tablet_manager.cc:1397] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:43:20.943369 2395 tablet_bootstrap.cc:492] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec: No bootstrap required, opened a new log
I20260926 14:43:20.943549 2395 ts_tablet_manager.cc:1397] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:43:20.943876 2396 tablet_bootstrap.cc:492] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f: No bootstrap required, opened a new log
I20260926 14:43:20.944056 2396 ts_tablet_manager.cc:1397] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260926 14:43:20.946108 2394 raft_consensus.cc:348] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dc473b4dc242169d6b1cf6d8b673ec" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33949 } } peers { permanent_uuid: "67e1405f49454a4488649893edda9e5f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43169 } } peers { permanent_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34997 } }
I20260926 14:43:20.946328 2394 raft_consensus.cc:374] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:20.946379 2394 raft_consensus.cc:729] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a1cb3db86b4bd4981a5a7b3095ff0c, State: Initialized, Role: FOLLOWER
I20260926 14:43:20.946786 2394 consensus_queue.cc:260] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dc473b4dc242169d6b1cf6d8b673ec" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33949 } } peers { permanent_uuid: "67e1405f49454a4488649893edda9e5f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43169 } } peers { permanent_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34997 } }
I20260926 14:43:20.947623 2394 ts_tablet_manager.cc:1428] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:43:20.947618 2395 raft_consensus.cc:348] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dc473b4dc242169d6b1cf6d8b673ec" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33949 } } peers { permanent_uuid: "67e1405f49454a4488649893edda9e5f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43169 } } peers { permanent_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34997 } }
I20260926 14:43:20.947618 2396 raft_consensus.cc:348] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dc473b4dc242169d6b1cf6d8b673ec" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33949 } } peers { permanent_uuid: "67e1405f49454a4488649893edda9e5f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43169 } } peers { permanent_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34997 } }
I20260926 14:43:20.947882 2395 raft_consensus.cc:374] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:20.947882 2396 raft_consensus.cc:374] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:20.947929 2395 raft_consensus.cc:729] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13dc473b4dc242169d6b1cf6d8b673ec, State: Initialized, Role: FOLLOWER
I20260926 14:43:20.947939 2396 raft_consensus.cc:729] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e1405f49454a4488649893edda9e5f, State: Initialized, Role: FOLLOWER
I20260926 14:43:20.947975 2113 heartbeater.cc:503] Master 127.16.90.254:39443 was elected leader, sending a full tablet report...
I20260926 14:43:20.948335 2396 consensus_queue.cc:260] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dc473b4dc242169d6b1cf6d8b673ec" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33949 } } peers { permanent_uuid: "67e1405f49454a4488649893edda9e5f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43169 } } peers { permanent_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34997 } }
I20260926 14:43:20.948335 2395 consensus_queue.cc:260] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dc473b4dc242169d6b1cf6d8b673ec" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33949 } } peers { permanent_uuid: "67e1405f49454a4488649893edda9e5f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43169 } } peers { permanent_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34997 } }
I20260926 14:43:20.948949 2377 heartbeater.cc:503] Master 127.16.90.254:39443 was elected leader, sending a full tablet report...
I20260926 14:43:20.949177 2244 heartbeater.cc:503] Master 127.16.90.254:39443 was elected leader, sending a full tablet report...
I20260926 14:43:20.949286 2395 ts_tablet_manager.cc:1428] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:43:20.950421 2396 ts_tablet_manager.cc:1428] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
W20260926 14:43:20.955300 2245 tablet.cc:2369] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:43:21.027019 2114 tablet.cc:2369] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:43:21.135808 2378 tablet.cc:2369] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:43:21.183364 2400 raft_consensus.cc:482] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:43:21.183580 2400 raft_consensus.cc:504] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dc473b4dc242169d6b1cf6d8b673ec" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33949 } } peers { permanent_uuid: "67e1405f49454a4488649893edda9e5f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43169 } } peers { permanent_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34997 } }
I20260926 14:43:21.184729 2400 leader_election.cc:288] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13dc473b4dc242169d6b1cf6d8b673ec (127.16.90.195:33949), 67e1405f49454a4488649893edda9e5f (127.16.90.194:43169)
I20260926 14:43:21.188565 2403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:21.184966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:33949 (local address 127.16.90.193:53315)
0926 14:43:21.185218 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:21.185234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:21.185365 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:21.185634 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:21.185636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:21.185658 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:21.185792 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:21.185798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:21.186587 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:21.186590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:21.187463 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:21.187472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:21.188138 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:21.188141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:21.188240 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:21.188278 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 14:43:21.188347 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:43:21.188402 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":4,"spinlock_wait_cycles":16512,"thread_start_us":75,"threads_started":1}
I20260926 14:43:21.188570 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:21.185097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:53315 (local address 127.16.90.195:33949)
0926 14:43:21.185219 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:21.185223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:21.185325 (+ 102us) server_negotiation.cc:407] Connection header received
0926 14:43:21.185469 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:21.185472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:21.185525 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:21.185600 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:21.185902 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:21.186494 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:21.187587 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:21.188036 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:21.188131 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:21.188169 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:21.188372 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 14:43:21.188443 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:43:21.188619 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:21.185478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:42203 (local address 127.16.90.194:43169)
0926 14:43:21.185595 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:21.185599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:21.185690 (+ 91us) server_negotiation.cc:407] Connection header received
0926 14:43:21.185815 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:21.185818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:21.185868 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:21.185936 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:21.186190 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:21.186716 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:21.187754 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:21.188131 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:21.188242 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:21.188285 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:21.188449 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 14:43:21.188517 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:43:21.188622 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:21.185326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:43169 (local address 127.16.90.193:42203)
0926 14:43:21.185613 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:21.185625 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:21.185726 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:21.185958 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:21.185961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:21.185978 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:21.186095 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:21.186102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:21.186829 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:21.186831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:21.187615 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:21.187624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:21.188250 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:21.188253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:21.188330 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:21.188353 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:43:21.188424 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:43:21.188484 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"spinlock_wait_cycles":384,"thread_start_us":73,"threads_started":1}
I20260926 14:43:21.189214 2198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f6f8b6f604bcfa36ae63a27f9d88d" candidate_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e1405f49454a4488649893edda9e5f" is_pre_election: true
I20260926 14:43:21.189237 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f6f8b6f604bcfa36ae63a27f9d88d" candidate_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13dc473b4dc242169d6b1cf6d8b673ec" is_pre_election: true
I20260926 14:43:21.189479 2198 raft_consensus.cc:2454] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a1cb3db86b4bd4981a5a7b3095ff0c in term 0.
I20260926 14:43:21.189484 2331 raft_consensus.cc:2454] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a1cb3db86b4bd4981a5a7b3095ff0c in term 0.
I20260926 14:43:21.189889 2000 leader_election.cc:302] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13dc473b4dc242169d6b1cf6d8b673ec, 18a1cb3db86b4bd4981a5a7b3095ff0c; no voters:
I20260926 14:43:21.190140 2400 raft_consensus.cc:2781] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:43:21.190213 2400 raft_consensus.cc:482] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:43:21.190241 2400 raft_consensus.cc:3037] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:21.191201 2400 raft_consensus.cc:504] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dc473b4dc242169d6b1cf6d8b673ec" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33949 } } peers { permanent_uuid: "67e1405f49454a4488649893edda9e5f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43169 } } peers { permanent_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34997 } }
I20260926 14:43:21.191511 2400 leader_election.cc:288] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [CANDIDATE]: Term 1 election: Requested vote from peers 13dc473b4dc242169d6b1cf6d8b673ec (127.16.90.195:33949), 67e1405f49454a4488649893edda9e5f (127.16.90.194:43169)
I20260926 14:43:21.191838 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f6f8b6f604bcfa36ae63a27f9d88d" candidate_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13dc473b4dc242169d6b1cf6d8b673ec"
I20260926 14:43:21.191869 2198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f6f8b6f604bcfa36ae63a27f9d88d" candidate_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e1405f49454a4488649893edda9e5f"
I20260926 14:43:21.191938 2331 raft_consensus.cc:3037] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:21.191972 2198 raft_consensus.cc:3037] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:21.192873 2331 raft_consensus.cc:2454] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a1cb3db86b4bd4981a5a7b3095ff0c in term 1.
I20260926 14:43:21.192929 2198 raft_consensus.cc:2454] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a1cb3db86b4bd4981a5a7b3095ff0c in term 1.
I20260926 14:43:21.193141 2000 leader_election.cc:302] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13dc473b4dc242169d6b1cf6d8b673ec, 18a1cb3db86b4bd4981a5a7b3095ff0c; no voters:
I20260926 14:43:21.193315 2400 raft_consensus.cc:2781] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:21.193495 2400 raft_consensus.cc:686] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [term 1 LEADER]: Becoming Leader. State: Replica: 18a1cb3db86b4bd4981a5a7b3095ff0c, State: Running, Role: LEADER
I20260926 14:43:21.193812 2400 consensus_queue.cc:237] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dc473b4dc242169d6b1cf6d8b673ec" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33949 } } peers { permanent_uuid: "67e1405f49454a4488649893edda9e5f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43169 } } peers { permanent_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34997 } }
I20260926 14:43:21.196928 1923 catalog_manager.cc:5495] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 18a1cb3db86b4bd4981a5a7b3095ff0c (127.16.90.193). New cstate: current_term: 1 leader_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dc473b4dc242169d6b1cf6d8b673ec" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e1405f49454a4488649893edda9e5f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18a1cb3db86b4bd4981a5a7b3095ff0c" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 34997 } health_report { overall_health: HEALTHY } } }
I20260926 14:43:21.216974 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:21.213685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.16.90.193:34997)
0926 14:43:21.213800 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:21.213804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:21.213832 (+ 28us) server_negotiation.cc:407] Connection header received
0926 14:43:21.213902 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:21.213904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:21.213956 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:21.214031 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:21.214197 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:21.214747 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:21.215655 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:21.215906 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:21.215983 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:21.216260 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:21.216659 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:21.216744 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:21.216826 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 14:43:21.216870 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:43:21.222599 2331 raft_consensus.cc:1261] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec [term 1 FOLLOWER]: Refusing update from remote peer 18a1cb3db86b4bd4981a5a7b3095ff0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:21.222622 2198 raft_consensus.cc:1261] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f [term 1 FOLLOWER]: Refusing update from remote peer 18a1cb3db86b4bd4981a5a7b3095ff0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:21.223272 2400 consensus_queue.cc:1035] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13dc473b4dc242169d6b1cf6d8b673ec" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 33949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:21.223469 2405 consensus_queue.cc:1035] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e1405f49454a4488649893edda9e5f" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:21.230649 2409 mvcc.cc:205] Tried to move back new op lower bound from 7333616849800806400 to 7333616849692491776. Current Snapshot: MvccSnapshot[applied={T|T < 7333616849800806400}]
I20260926 14:43:21.232282 2410 mvcc.cc:205] Tried to move back new op lower bound from 7333616849800806400 to 7333616849692491776. Current Snapshot: MvccSnapshot[applied={T|T < 7333616849800806400}]
I20260926 14:43:21.238509 1921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:43:21.238652 1921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:43:21.240803 1921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e10030946d1491cbfa691916486a2d8: Sending DeleteTablet for 3 replicas of tablet bd9f6f8b6f604bcfa36ae63a27f9d88d
I20260926 14:43:21.241577 2309 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9f6f8b6f604bcfa36ae63a27f9d88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:21 UTC) from {username='slave'} at 127.0.0.1:51254
I20260926 14:43:21.241755 2047 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9f6f8b6f604bcfa36ae63a27f9d88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:21 UTC) from {username='slave'} at 127.0.0.1:50994
I20260926 14:43:21.242041 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 1985
I20260926 14:43:21.242239 2419 tablet_replica.cc:330] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 18a1cb3db86b4bd4981a5a7b3095ff0c: stopping tablet replica
I20260926 14:43:21.242225 2178 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9f6f8b6f604bcfa36ae63a27f9d88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:21 UTC) from {username='slave'} at 127.0.0.1:41244
I20260926 14:43:21.242655 2418 tablet_replica.cc:330] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec: stopping tablet replica
I20260926 14:43:21.242946 2418 raft_consensus.cc:2229] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:43:21.243229 2418 raft_consensus.cc:2258] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:43:21.244609 2418 ts_tablet_manager.cc:1905] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:43:21.246846 2418 ts_tablet_manager.cc:1918] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:43:21.246924 2418 log.cc:1196] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/bd9f6f8b6f604bcfa36ae63a27f9d88d
I20260926 14:43:21.247133 2420 tablet_replica.cc:330] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f: stopping tablet replica
I20260926 14:43:21.247165 2418 ts_tablet_manager.cc:1939] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 13dc473b4dc242169d6b1cf6d8b673ec: Deleting consensus metadata
I20260926 14:43:21.247467 2420 raft_consensus.cc:2229] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:43:21.247834 2420 raft_consensus.cc:2258] T bd9f6f8b6f604bcfa36ae63a27f9d88d P 67e1405f49454a4488649893edda9e5f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:43:21.248302 1911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:34997 (error 108)
I20260926 14:43:21.248374 1908 catalog_manager.cc:4841] TS 13dc473b4dc242169d6b1cf6d8b673ec (127.16.90.195:33949): tablet bd9f6f8b6f604bcfa36ae63a27f9d88d (table test-table [id=f21100d8f1d0433fbc301877b5ce7482]) successfully deleted
I20260926 14:43:21.248500 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 2116
I20260926 14:43:21.249588 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:21.249343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:34997 (local address 127.0.0.1:51006)
0926 14:43:21.249457 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:21.249519 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:43:21.249715 1911 catalog_manager.cc:4584] TS 18a1cb3db86b4bd4981a5a7b3095ff0c (127.16.90.193:34997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd9f6f8b6f604bcfa36ae63a27f9d88d: Network error: Client connection negotiation failed: client connection to 127.16.90.193:34997: connect: Connection refused (error 111)
I20260926 14:43:21.253571 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 2247
I20260926 14:43:21.254331 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:21.253753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:43169 (local address 127.0.0.1:41258)
0926 14:43:21.254231 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:21.254273 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.194:43169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419}
I20260926 14:43:21.258424 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 1893
2026-09-26T14:43:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1182 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:21 dist-test-slave-x4sn krb5kdc[2424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:21 dist-test-slave-x4sn krb5kdc[2424](info): set up 2 sockets
Sep 26 14:43:21 dist-test-slave-x4sn krb5kdc[2424](info): commencing operation
krb5kdc: starting...
W20260926 14:43:22.200914 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:22 dist-test-slave-x4sn krb5kdc[2424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:43:22.321761 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:33925
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:33925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:22.428380 2437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:22.428653 2437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:22.428709 2437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:22.432694 2437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:22.432756 2437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:22.432783 2437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:22.432804 2437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:22.432833 2437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:22.437966 2437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:33925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:33925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:22.439172 2437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:22.440294 2437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:22.448325 2442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:22.448398 2443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:22.448383 2437 server_base.cc:1034] running on GCE node
W20260926 14:43:22.448311 2445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:22.449110 2437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:22.450150 2437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:22.451292 2437 hybrid_clock.cc:648] HybridClock initialized: now 1790433802451270 us; error 38 us; skew 500 ppm
Sep 26 14:43:22 dist-test-slave-x4sn krb5kdc[2424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433802, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:22.453959 2437 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:43:22.455039 2437 webserver.cc:460] Webserver started at http://127.16.90.254:40441/ using document root <none> and password file <none>
I20260926 14:43:22.455724 2437 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:22.455796 2437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:22.456009 2437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:22.458576 2437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "2226052e5c4f4aed97f1241eb8114d84"
format_stamp: "Formatted at 2026-09-26 14:43:22 on dist-test-slave-x4sn"
server_key: "d3cc0064d52befcaa5b3a53a2381bd9b"
server_key_iv: "45365d68eac25d407701e36394cf6241"
server_key_version: "encryptionkey@0"
I20260926 14:43:22.459087 2437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "2226052e5c4f4aed97f1241eb8114d84"
format_stamp: "Formatted at 2026-09-26 14:43:22 on dist-test-slave-x4sn"
server_key: "d3cc0064d52befcaa5b3a53a2381bd9b"
server_key_iv: "45365d68eac25d407701e36394cf6241"
server_key_version: "encryptionkey@0"
I20260926 14:43:22.462590 2437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:43:22.465369 2452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:22.466703 2437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:43:22.466811 2437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "2226052e5c4f4aed97f1241eb8114d84"
format_stamp: "Formatted at 2026-09-26 14:43:22 on dist-test-slave-x4sn"
server_key: "d3cc0064d52befcaa5b3a53a2381bd9b"
server_key_iv: "45365d68eac25d407701e36394cf6241"
server_key_version: "encryptionkey@0"
I20260926 14:43:22.466920 2437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:22.476914 2437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:22.479797 2437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:22.479955 2437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:22.486359 2437 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:33925
I20260926 14:43:22.486403 2504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:33925 every 8 connection(s)
I20260926 14:43:22.487243 2437 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:22.489840 2505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:22.495144 2505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84: Bootstrap starting.
I20260926 14:43:22.496997 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 2437
I20260926 14:43:22.497136 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:22.497431 16747 external_mini_cluster.cc:1428] Setting key f9e62a4eff01c5e08f998f1009ab97b1
I20260926 14:43:22.497745 2505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:22.498615 2505 log.cc:824] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:22.500523 2505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84: No bootstrap required, opened a new log
I20260926 14:43:22.504587 2505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2226052e5c4f4aed97f1241eb8114d84" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33925 } }
I20260926 14:43:22.504873 2505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:22.504931 2505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2226052e5c4f4aed97f1241eb8114d84, State: Initialized, Role: FOLLOWER
I20260926 14:43:22.505429 2505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2226052e5c4f4aed97f1241eb8114d84" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33925 } }
I20260926 14:43:22.505599 2505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:22.505669 2505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:22.505764 2505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 14:43:22 dist-test-slave-x4sn krb5kdc[2424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:22.506654 2505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2226052e5c4f4aed97f1241eb8114d84" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33925 } }
I20260926 14:43:22.506959 2505 leader_election.cc:302] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2226052e5c4f4aed97f1241eb8114d84; no voters:
I20260926 14:43:22.507247 2505 leader_election.cc:288] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:22.507325 2510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:22.507509 2510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [term 1 LEADER]: Becoming Leader. State: Replica: 2226052e5c4f4aed97f1241eb8114d84, State: Running, Role: LEADER
I20260926 14:43:22.507825 2510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2226052e5c4f4aed97f1241eb8114d84" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33925 } }
I20260926 14:43:22.508551 2505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:22.508996 2510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2226052e5c4f4aed97f1241eb8114d84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2226052e5c4f4aed97f1241eb8114d84" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33925 } } }
I20260926 14:43:22.509138 2510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:22.509342 2510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2226052e5c4f4aed97f1241eb8114d84. Latest consensus state: current_term: 1 leader_uuid: "2226052e5c4f4aed97f1241eb8114d84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2226052e5c4f4aed97f1241eb8114d84" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33925 } } }
I20260926 14:43:22.509428 2510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:22.511585 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:22.498517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.16.90.254:33925)
0926 14:43:22.498906 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:22.498921 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:22.498962 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:43:22.499720 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:22.499739 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:22.500052 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:22.500322 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:22.501197 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:22.502149 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:22.503329 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:22.503819 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:22.503931 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:22.506565 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:22.506571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:22.506583 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:22.506609 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:22.508309 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:22.508739 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:22.508744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:22.508749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:22.508873 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:22.509174 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:22.509177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:22.509179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:22.509686 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:22.509873 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:22.510188 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 14:43:22.510489 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":123,"threads_started":1}
W20260926 14:43:22.513803 2525 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:43:22.513870 2525 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:43:22.513922 2515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:22.514571 2515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:22.519685 2515 catalog_manager.cc:1347] Generated new cluster ID: 782b0f45456b4a758c455bcdd4633c40
I20260926 14:43:22.519742 2515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:22.527818 2515 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:22.529021 2515 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:22.535218 2515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84: Generated new TSK 0
I20260926 14:43:22.536046 2515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:43:22.564445 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:33925
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:22.670367 2531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:22.670630 2531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:22.670673 2531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:22.674670 2531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:22.674726 2531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:22.674782 2531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:43:22.679893 2531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:33925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:22.680836 2531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:22.681857 2531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:22.690647 2537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:22.690764 2536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:22.690647 2539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:22.690933 2531 server_base.cc:1034] running on GCE node
I20260926 14:43:22.691306 2531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:22.691924 2531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:22.693089 2531 hybrid_clock.cc:648] HybridClock initialized: now 1790433802693067 us; error 27 us; skew 500 ppm
Sep 26 14:43:22 dist-test-slave-x4sn krb5kdc[2424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433802, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:22.695528 2531 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:43:22.696350 2531 webserver.cc:460] Webserver started at http://127.16.90.193:36927/ using document root <none> and password file <none>
I20260926 14:43:22.696852 2531 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:22.696915 2531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:22.697106 2531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:22.699025 2531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "3256f4d6be2341febc82fc2eaaa6aff6"
format_stamp: "Formatted at 2026-09-26 14:43:22 on dist-test-slave-x4sn"
server_key: "9a6984ca877b2f4a1616b3a323740782"
server_key_iv: "224290399663b3133caab8f899cf8a85"
server_key_version: "encryptionkey@0"
I20260926 14:43:22.699540 2531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "3256f4d6be2341febc82fc2eaaa6aff6"
format_stamp: "Formatted at 2026-09-26 14:43:22 on dist-test-slave-x4sn"
server_key: "9a6984ca877b2f4a1616b3a323740782"
server_key_iv: "224290399663b3133caab8f899cf8a85"
server_key_version: "encryptionkey@0"
I20260926 14:43:22.702960 2531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260926 14:43:22.705189 2546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:22.706315 2531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:43:22.706418 2531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "3256f4d6be2341febc82fc2eaaa6aff6"
format_stamp: "Formatted at 2026-09-26 14:43:22 on dist-test-slave-x4sn"
server_key: "9a6984ca877b2f4a1616b3a323740782"
server_key_iv: "224290399663b3133caab8f899cf8a85"
server_key_version: "encryptionkey@0"
I20260926 14:43:22.706501 2531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:22.727866 2531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:22.730546 2531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:22.730728 2531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:22.731320 2531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:22.732163 2531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:22.732208 2531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:22.732259 2531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:22.732286 2531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:22.740659 2531 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:42913
I20260926 14:43:22.740688 2660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:42913 every 8 connection(s)
I20260926 14:43:22.741526 2531 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:43:22.750319 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 2531
I20260926 14:43:22.750452 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:43:22.750756 16747 external_mini_cluster.cc:1428] Setting key b043aee0ad5105603c3c9989095e2da8
Sep 26 14:43:22 dist-test-slave-x4sn krb5kdc[2424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433802, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:22.754673 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:22.743207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:35641 (local address 127.16.90.254:33925)
0926 14:43:22.743327 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:22.743330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:22.744197 (+ 867us) server_negotiation.cc:407] Connection header received
0926 14:43:22.745509 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:22.745513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:22.745571 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:22.745643 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:22.746793 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:22.747336 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:22.748478 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:22.748730 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:22.748820 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:22.751381 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:22.751385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:22.751387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:22.751412 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:22.752803 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:22.753262 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:22.753265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:22.753267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:22.753342 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:22.753642 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:22.753646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:22.753647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:22.753826 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:22.753909 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:22.754383 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 14:43:22.754503 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:43:22.755445 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:22.743583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:33925 (local address 127.16.90.193:35641)
0926 14:43:22.744045 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:22.744087 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:22.745299 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:22.745822 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:22.745828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:22.746167 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:22.746634 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:22.746648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:22.747454 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:22.747457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:22.748359 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:22.748368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:22.748868 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:22.748873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:22.749029 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:22.749556 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:22.749572 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:22.751222 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:22.752905 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:22.752910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:22.752924 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:22.753174 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:22.753422 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:22.753425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:22.753427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:22.753559 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:22.753921 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:22.753928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:22.754167 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 14:43:22.754328 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 14:43:22.754635 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":116,"threads_started":1}
I20260926 14:43:22.756803 2661 heartbeater.cc:347] Connected to a master server at 127.16.90.254:33925
I20260926 14:43:22.757058 2661 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:22.757573 2661 heartbeater.cc:511] Master 127.16.90.254:33925 requested a full tablet report, sending...
I20260926 14:43:22.759303 2469 ts_manager.cc:194] Registered new tserver with Master: 3256f4d6be2341febc82fc2eaaa6aff6 (127.16.90.193:42913)
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
I20260926 14:43:22.760602 2469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:35641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:43:22.776029 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:33925
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:22.882118 2666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:22.882356 2666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:22.882402 2666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:22.886312 2666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:22.886366 2666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:22.886438 2666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:43:22.891502 2666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:33925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:22.892504 2666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:22.893461 2666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:22.902395 2671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:22.902575 2666 server_base.cc:1034] running on GCE node
W20260926 14:43:22.902411 2672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:22.902390 2674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:22.902981 2666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:22.903601 2666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:22.904754 2666 hybrid_clock.cc:648] HybridClock initialized: now 1790433802904739 us; error 26 us; skew 500 ppm
Sep 26 14:43:22 dist-test-slave-x4sn krb5kdc[2424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433802, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:22.907231 2666 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:43:22.908085 2666 webserver.cc:460] Webserver started at http://127.16.90.194:40381/ using document root <none> and password file <none>
I20260926 14:43:22.908612 2666 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:22.908680 2666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:22.908908 2666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:22.911083 2666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "372e62af0e6546ee97a11facfe56201c"
format_stamp: "Formatted at 2026-09-26 14:43:22 on dist-test-slave-x4sn"
server_key: "bd5aca045d0215ec9ae1771ac2429b9a"
server_key_iv: "2ad2e1985c46e9889a2ada40fd6cd249"
server_key_version: "encryptionkey@0"
I20260926 14:43:22.911603 2666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "372e62af0e6546ee97a11facfe56201c"
format_stamp: "Formatted at 2026-09-26 14:43:22 on dist-test-slave-x4sn"
server_key: "bd5aca045d0215ec9ae1771ac2429b9a"
server_key_iv: "2ad2e1985c46e9889a2ada40fd6cd249"
server_key_version: "encryptionkey@0"
I20260926 14:43:22.915155 2666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:22.917408 2681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:22.918605 2666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:22.918715 2666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "372e62af0e6546ee97a11facfe56201c"
format_stamp: "Formatted at 2026-09-26 14:43:22 on dist-test-slave-x4sn"
server_key: "bd5aca045d0215ec9ae1771ac2429b9a"
server_key_iv: "2ad2e1985c46e9889a2ada40fd6cd249"
server_key_version: "encryptionkey@0"
I20260926 14:43:22.918804 2666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:22.937086 2666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:22.939858 2666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:22.940029 2666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:22.940598 2666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:22.941421 2666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:22.941468 2666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:22.941520 2666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:22.941545 2666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:22.949676 2666 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:37127
I20260926 14:43:22.949774 2795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:37127 every 8 connection(s)
I20260926 14:43:22.950538 2666 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:43:22.951529 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 2666
I20260926 14:43:22.951656 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:43:22.951977 16747 external_mini_cluster.cc:1428] Setting key 9770e02e77283fc6b0cb5d30e868b1b0
Sep 26 14:43:22 dist-test-slave-x4sn krb5kdc[2424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433802, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
I20260926 14:43:22.964250 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:22.952398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:55417 (local address 127.16.90.254:33925)
0926 14:43:22.952528 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:22.952533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:22.953509 (+ 976us) server_negotiation.cc:407] Connection header received
0926 14:43:22.955179 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:22.955181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:22.955232 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:22.955311 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:22.956469 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:22.957216 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:22.958351 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:22.958688 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:22.958807 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:22.961146 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:22.961149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:22.961151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:22.961180 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:22.962430 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:22.962855 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:22.962857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:22.962858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:22.962926 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:22.963210 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:22.963211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:22.963212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:22.963374 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:22.963449 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:22.964025 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 14:43:22.964126 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:43:22.965310 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:22.952854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:33925 (local address 127.16.90.194:55417)
0926 14:43:22.953378 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:22.953421 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:22.954946 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:22.955500 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:22.955507 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:22.955888 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:22.956349 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:22.956361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:22.957314 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:22.957317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:22.958232 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:22.958242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:22.958820 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:22.958822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:22.958952 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:22.959460 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:22.959476 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:22.960984 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:22.962523 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:22.962527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:22.962538 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:22.962776 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:22.962989 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:22.962992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:22.962993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:22.963137 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:22.963463 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:22.963467 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:22.963769 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 14:43:22.963981 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 14:43:22.964321 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":147,"threads_started":1}
I20260926 14:43:22.966377 2796 heartbeater.cc:347] Connected to a master server at 127.16.90.254:33925
I20260926 14:43:22.966617 2796 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:22.967151 2796 heartbeater.cc:511] Master 127.16.90.254:33925 requested a full tablet report, sending...
I20260926 14:43:22.968245 2469 ts_manager.cc:194] Registered new tserver with Master: 372e62af0e6546ee97a11facfe56201c (127.16.90.194:37127)
I20260926 14:43:22.968801 2469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:55417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:43:22.978235 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:33925
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:23.086632 2801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:23.086884 2801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:23.086931 2801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:23.090878 2801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:23.090934 2801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:23.091003 2801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:43:23.096292 2801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:33925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:23.097294 2801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:23.098263 2801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:23.107273 2809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:23.107280 2806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:23.107415 2801 server_base.cc:1034] running on GCE node
W20260926 14:43:23.107311 2807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:23.108028 2801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:23.108660 2801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:23.109818 2801 hybrid_clock.cc:648] HybridClock initialized: now 1790433803109778 us; error 49 us; skew 500 ppm
Sep 26 14:43:23 dist-test-slave-x4sn krb5kdc[2424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:23.112481 2801 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:43:23.113263 2801 webserver.cc:460] Webserver started at http://127.16.90.195:39315/ using document root <none> and password file <none>
I20260926 14:43:23.113768 2801 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:23.113826 2801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:23.113987 2801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:23.115998 2801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "4e1623375e1346c5ae95c825ff94401a"
format_stamp: "Formatted at 2026-09-26 14:43:23 on dist-test-slave-x4sn"
server_key: "81df65358593b05439e9618ea734369f"
server_key_iv: "4902a75db8617994059f94676ad74889"
server_key_version: "encryptionkey@0"
I20260926 14:43:23.116496 2801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "4e1623375e1346c5ae95c825ff94401a"
format_stamp: "Formatted at 2026-09-26 14:43:23 on dist-test-slave-x4sn"
server_key: "81df65358593b05439e9618ea734369f"
server_key_iv: "4902a75db8617994059f94676ad74889"
server_key_version: "encryptionkey@0"
I20260926 14:43:23.120124 2801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:23.122530 2816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:23.123732 2801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:23.123835 2801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "4e1623375e1346c5ae95c825ff94401a"
format_stamp: "Formatted at 2026-09-26 14:43:23 on dist-test-slave-x4sn"
server_key: "81df65358593b05439e9618ea734369f"
server_key_iv: "4902a75db8617994059f94676ad74889"
server_key_version: "encryptionkey@0"
I20260926 14:43:23.123926 2801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:23.167462 2801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:23.170720 2801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:23.170898 2801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:23.171540 2801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:23.172407 2801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:23.172456 2801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:23.172509 2801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:23.172537 2801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:23.181190 2801 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:44515
I20260926 14:43:23.181203 2929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:44515 every 8 connection(s)
I20260926 14:43:23.182094 2801 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:43:23.184779 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 2801
I20260926 14:43:23.184901 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:43:23.185182 16747 external_mini_cluster.cc:1428] Setting key abf54f1fafb99a7e13c34ba48d1e1cb5
Sep 26 14:43:23 dist-test-slave-x4sn krb5kdc[2424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433803, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:23.194924 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.183812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:43455 (local address 127.16.90.254:33925)
0926 14:43:23.183951 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:23.183955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:23.184857 (+ 902us) server_negotiation.cc:407] Connection header received
0926 14:43:23.186323 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:23.186326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:23.186378 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:23.186450 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:23.187558 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.188123 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.189221 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.189433 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.189518 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.191792 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:23.191796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:23.191798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:23.191822 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:23.193087 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:23.193541 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:23.193544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:23.193545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:23.193617 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:23.193914 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:23.193918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:23.193919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:23.194085 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:23.194176 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:23.194635 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 14:43:23.194764 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:43:23.195737 2932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.184201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:33925 (local address 127.16.90.195:43455)
0926 14:43:23.184699 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:23.184742 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:23.186134 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:23.186619 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:23.186626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:23.186963 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:23.187430 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.187443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.188241 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.188244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.189110 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.189118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.189560 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.189564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.189729 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.190258 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:23.190274 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:23.191615 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:23.193191 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:23.193197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:23.193211 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:23.193453 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:23.193700 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:23.193703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:23.193705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:23.193832 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:23.194177 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:23.194183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:23.194416 (+ 233us) client_negotiation.cc:769] Sending connection context
0926 14:43:23.194578 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 14:43:23.194896 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260926 14:43:23.196720 2930 heartbeater.cc:347] Connected to a master server at 127.16.90.254:33925
I20260926 14:43:23.196942 2930 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:23.197438 2930 heartbeater.cc:511] Master 127.16.90.254:33925 requested a full tablet report, sending...
I20260926 14:43:23.198434 2469 ts_manager.cc:194] Registered new tserver with Master: 4e1623375e1346c5ae95c825ff94401a (127.16.90.195:44515)
I20260926 14:43:23.198974 2469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:43455
I20260926 14:43:23.210474 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:43:23.219424 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.212428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.16.90.254:33925)
0926 14:43:23.212542 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:23.212546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:23.212692 (+ 146us) server_negotiation.cc:407] Connection header received
0926 14:43:23.212812 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:23.212815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:23.212861 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:23.212924 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:23.213581 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.214098 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.214906 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.215130 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.215222 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.216760 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:23.216763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:23.216765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:23.216790 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:23.218037 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:23.218442 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:23.218444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:23.218445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:23.218515 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:23.218750 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:23.218752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:23.218753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:23.218935 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:23.219043 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:23.219232 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 14:43:23.219306 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:43:23.222301 2469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:43:23.225411 2469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:43:23.239543 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.233488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.16.90.193:42913)
0926 14:43:23.234733 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:23.234750 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:23.234773 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:43:23.234847 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:23.234853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:23.235046 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:23.235206 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:23.235340 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.236510 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.237968 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.238800 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.239013 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.239142 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:23.239284 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 14:43:23.239382 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":64,"threads_started":1}
I20260926 14:43:23.239549 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.234103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.16.90.194:37127)
0926 14:43:23.234539 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:23.234551 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:23.234574 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:43:23.234642 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:23.234646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:23.234842 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:23.235316 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:23.235458 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.236242 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.237972 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.238808 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.239151 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.239286 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:23.239333 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:43:23.239415 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":150,"threads_started":1}
I20260926 14:43:23.239545 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.233332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:42913 (local address 127.0.0.1:54802)
0926 14:43:23.233703 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:23.233754 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:23.233880 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:23.235181 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:23.235184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:23.235199 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:23.235271 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.235276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.236636 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.236640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.237838 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.237847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.238940 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.238944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.239084 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.239131 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 14:43:23.239252 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 14:43:23.239404 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":80,"thread_start_us":71,"threads_started":1}
I20260926 14:43:23.239547 2943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.233594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:37127 (local address 127.0.0.1:55978)
0926 14:43:23.233827 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:23.233838 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:23.233903 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:23.235013 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:23.235018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:23.235043 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:23.235151 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.235167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.236360 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.236364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.237837 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.237848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.238945 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.238949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.239072 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.239199 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 14:43:23.239261 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 14:43:23.239409 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":179,"thread_start_us":106,"threads_started":1}
I20260926 14:43:23.240214 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.234033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:44515 (local address 127.0.0.1:47972)
0926 14:43:23.234268 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:23.234280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:23.234359 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:23.235359 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:23.235361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:23.235375 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:23.235442 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.235447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.236769 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.236772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.237823 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.237832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.238940 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.238944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.239963 (+ 1019us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.239988 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:43:23.240053 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:43:23.240114 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":31,"thread_start_us":90,"threads_started":1}
I20260926 14:43:23.242693 2730 tablet_service.cc:1467] Processing CreateTablet for tablet 82535263cd81422294d0f8d739221d89 (DEFAULT_TABLE table=test-table [id=dc04574f26404c66b41f7c9ca3301fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:23.242693 2595 tablet_service.cc:1467] Processing CreateTablet for tablet 82535263cd81422294d0f8d739221d89 (DEFAULT_TABLE table=test-table [id=dc04574f26404c66b41f7c9ca3301fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:23.243836 2595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82535263cd81422294d0f8d739221d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:23.243836 2730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82535263cd81422294d0f8d739221d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:23.243891 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.234349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.16.90.195:44515)
0926 14:43:23.234809 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:23.234820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:23.234838 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:43:23.234906 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:23.234910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:23.235046 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:23.235206 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:23.235547 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.236645 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.237938 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.238800 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.238992 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.239142 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:23.240448 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0926 14:43:23.243784 (+ 3336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":89,"threads_started":1}
I20260926 14:43:23.246585 2864 tablet_service.cc:1467] Processing CreateTablet for tablet 82535263cd81422294d0f8d739221d89 (DEFAULT_TABLE table=test-table [id=dc04574f26404c66b41f7c9ca3301fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:23.247612 2864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82535263cd81422294d0f8d739221d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:23.251142 2947 tablet_bootstrap.cc:492] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6: Bootstrap starting.
I20260926 14:43:23.253183 2948 tablet_bootstrap.cc:492] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c: Bootstrap starting.
I20260926 14:43:23.253533 2947 tablet_bootstrap.cc:654] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:23.254122 2949 tablet_bootstrap.cc:492] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a: Bootstrap starting.
I20260926 14:43:23.255019 2947 log.cc:824] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:23.255262 2948 tablet_bootstrap.cc:654] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:23.256064 2949 tablet_bootstrap.cc:654] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:23.256476 2948 log.cc:824] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:23.257206 2949 log.cc:824] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:23.257251 2947 tablet_bootstrap.cc:492] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6: No bootstrap required, opened a new log
I20260926 14:43:23.257426 2947 ts_tablet_manager.cc:1397] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:43:23.258086 2948 tablet_bootstrap.cc:492] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c: No bootstrap required, opened a new log
I20260926 14:43:23.258273 2948 ts_tablet_manager.cc:1397] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:43:23.259051 2949 tablet_bootstrap.cc:492] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a: No bootstrap required, opened a new log
I20260926 14:43:23.259218 2949 ts_tablet_manager.cc:1397] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:43:23.260368 2947 raft_consensus.cc:348] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42913 } } peers { permanent_uuid: "4e1623375e1346c5ae95c825ff94401a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44515 } } peers { permanent_uuid: "372e62af0e6546ee97a11facfe56201c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37127 } }
I20260926 14:43:23.260579 2947 raft_consensus.cc:374] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:23.260632 2947 raft_consensus.cc:729] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3256f4d6be2341febc82fc2eaaa6aff6, State: Initialized, Role: FOLLOWER
I20260926 14:43:23.261010 2947 consensus_queue.cc:260] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42913 } } peers { permanent_uuid: "4e1623375e1346c5ae95c825ff94401a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44515 } } peers { permanent_uuid: "372e62af0e6546ee97a11facfe56201c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37127 } }
I20260926 14:43:23.261559 2661 heartbeater.cc:503] Master 127.16.90.254:33925 was elected leader, sending a full tablet report...
I20260926 14:43:23.261920 2947 ts_tablet_manager.cc:1428] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 14:43:23.261900 2948 raft_consensus.cc:348] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42913 } } peers { permanent_uuid: "4e1623375e1346c5ae95c825ff94401a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44515 } } peers { permanent_uuid: "372e62af0e6546ee97a11facfe56201c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37127 } }
I20260926 14:43:23.262171 2948 raft_consensus.cc:374] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:23.262233 2948 raft_consensus.cc:729] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372e62af0e6546ee97a11facfe56201c, State: Initialized, Role: FOLLOWER
I20260926 14:43:23.262640 2948 consensus_queue.cc:260] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42913 } } peers { permanent_uuid: "4e1623375e1346c5ae95c825ff94401a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44515 } } peers { permanent_uuid: "372e62af0e6546ee97a11facfe56201c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37127 } }
I20260926 14:43:23.263257 2796 heartbeater.cc:503] Master 127.16.90.254:33925 was elected leader, sending a full tablet report...
I20260926 14:43:23.263576 2948 ts_tablet_manager.cc:1428] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:43:23.263752 2949 raft_consensus.cc:348] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42913 } } peers { permanent_uuid: "4e1623375e1346c5ae95c825ff94401a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44515 } } peers { permanent_uuid: "372e62af0e6546ee97a11facfe56201c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37127 } }
I20260926 14:43:23.263942 2949 raft_consensus.cc:374] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:23.263974 2949 raft_consensus.cc:729] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1623375e1346c5ae95c825ff94401a, State: Initialized, Role: FOLLOWER
I20260926 14:43:23.264341 2949 consensus_queue.cc:260] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42913 } } peers { permanent_uuid: "4e1623375e1346c5ae95c825ff94401a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44515 } } peers { permanent_uuid: "372e62af0e6546ee97a11facfe56201c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37127 } }
I20260926 14:43:23.264926 2930 heartbeater.cc:503] Master 127.16.90.254:33925 was elected leader, sending a full tablet report...
I20260926 14:43:23.265223 2949 ts_tablet_manager.cc:1428] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:43:23.382566 2953 raft_consensus.cc:482] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:43:23.382808 2953 raft_consensus.cc:504] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42913 } } peers { permanent_uuid: "4e1623375e1346c5ae95c825ff94401a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44515 } } peers { permanent_uuid: "372e62af0e6546ee97a11facfe56201c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37127 } }
I20260926 14:43:23.383991 2953 leader_election.cc:288] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e1623375e1346c5ae95c825ff94401a (127.16.90.195:44515), 372e62af0e6546ee97a11facfe56201c (127.16.90.194:37127)
I20260926 14:43:23.388931 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.384353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:37563 (local address 127.16.90.194:37127)
0926 14:43:23.384484 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:23.384488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:23.384725 (+ 237us) server_negotiation.cc:407] Connection header received
0926 14:43:23.384803 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:23.384806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:23.384851 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:23.384923 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:23.385258 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.386021 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.387714 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.388322 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.388439 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.388488 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:23.388743 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 14:43:23.388806 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 14:43:23.388931 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.384284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:50227 (local address 127.16.90.195:44515)
0926 14:43:23.384405 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:23.384409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:23.384676 (+ 267us) server_negotiation.cc:407] Connection header received
0926 14:43:23.384783 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:23.384786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:23.384841 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:23.384925 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:23.385261 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.386043 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.387714 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.388322 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.388439 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.388488 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:23.388742 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 14:43:23.388811 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:43:23.389020 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.384233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:37127 (local address 127.16.90.193:37563)
0926 14:43:23.384645 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:23.384657 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:23.384733 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:23.384937 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:23.384940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:23.384971 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:23.385154 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.385160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.386135 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.386138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.387560 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.387566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.388448 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.388452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.388601 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.388654 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 14:43:23.388719 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:43:23.388824 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":164,"spinlock_wait_cycles":8320,"thread_start_us":135,"threads_started":1}
I20260926 14:43:23.389024 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.384206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:44515 (local address 127.16.90.193:50227)
0926 14:43:23.384557 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:23.384571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:23.384687 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:23.384937 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:23.384940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:23.384965 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:23.385154 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.385160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.386135 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.386138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.387554 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.387566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.388448 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.388452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.388585 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.388644 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 14:43:23.388719 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 14:43:23.388824 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":197,"spinlock_wait_cycles":4352,"thread_start_us":137,"threads_started":1}
I20260926 14:43:23.389719 2750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82535263cd81422294d0f8d739221d89" candidate_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372e62af0e6546ee97a11facfe56201c" is_pre_election: true
I20260926 14:43:23.389722 2884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82535263cd81422294d0f8d739221d89" candidate_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e1623375e1346c5ae95c825ff94401a" is_pre_election: true
I20260926 14:43:23.389989 2884 raft_consensus.cc:2454] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3256f4d6be2341febc82fc2eaaa6aff6 in term 0.
I20260926 14:43:23.389992 2750 raft_consensus.cc:2454] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3256f4d6be2341febc82fc2eaaa6aff6 in term 0.
I20260926 14:43:23.390425 2547 leader_election.cc:302] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3256f4d6be2341febc82fc2eaaa6aff6, 4e1623375e1346c5ae95c825ff94401a; no voters:
I20260926 14:43:23.390695 2953 raft_consensus.cc:2781] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:43:23.390770 2953 raft_consensus.cc:482] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:43:23.390798 2953 raft_consensus.cc:3037] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:23.391853 2953 raft_consensus.cc:504] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42913 } } peers { permanent_uuid: "4e1623375e1346c5ae95c825ff94401a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44515 } } peers { permanent_uuid: "372e62af0e6546ee97a11facfe56201c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37127 } }
I20260926 14:43:23.392150 2953 leader_election.cc:288] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [CANDIDATE]: Term 1 election: Requested vote from peers 4e1623375e1346c5ae95c825ff94401a (127.16.90.195:44515), 372e62af0e6546ee97a11facfe56201c (127.16.90.194:37127)
I20260926 14:43:23.392488 2884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82535263cd81422294d0f8d739221d89" candidate_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e1623375e1346c5ae95c825ff94401a"
I20260926 14:43:23.392508 2750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82535263cd81422294d0f8d739221d89" candidate_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372e62af0e6546ee97a11facfe56201c"
I20260926 14:43:23.392588 2884 raft_consensus.cc:3037] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:23.392621 2750 raft_consensus.cc:3037] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:23.393571 2884 raft_consensus.cc:2454] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3256f4d6be2341febc82fc2eaaa6aff6 in term 1.
I20260926 14:43:23.393618 2750 raft_consensus.cc:2454] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3256f4d6be2341febc82fc2eaaa6aff6 in term 1.
I20260926 14:43:23.393842 2547 leader_election.cc:302] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3256f4d6be2341febc82fc2eaaa6aff6, 4e1623375e1346c5ae95c825ff94401a; no voters:
I20260926 14:43:23.394048 2953 raft_consensus.cc:2781] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:23.394249 2953 raft_consensus.cc:686] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 1 LEADER]: Becoming Leader. State: Replica: 3256f4d6be2341febc82fc2eaaa6aff6, State: Running, Role: LEADER
I20260926 14:43:23.394577 2953 consensus_queue.cc:237] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42913 } } peers { permanent_uuid: "4e1623375e1346c5ae95c825ff94401a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44515 } } peers { permanent_uuid: "372e62af0e6546ee97a11facfe56201c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37127 } }
I20260926 14:43:23.397758 2469 catalog_manager.cc:5495] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3256f4d6be2341febc82fc2eaaa6aff6 (127.16.90.193). New cstate: current_term: 1 leader_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3256f4d6be2341febc82fc2eaaa6aff6" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e1623375e1346c5ae95c825ff94401a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372e62af0e6546ee97a11facfe56201c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37127 } health_report { overall_health: UNKNOWN } } }
I20260926 14:43:23.427106 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.424057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.16.90.193:42913)
0926 14:43:23.424174 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:23.424177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:23.424191 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:43:23.424259 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:23.424261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:23.424313 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:23.424394 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:23.424599 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.425112 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.425840 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.426038 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.426119 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.426330 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:23.426716 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:23.426819 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:23.426921 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 14:43:23.426961 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:43:23.432855 2750 raft_consensus.cc:1261] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c [term 1 FOLLOWER]: Refusing update from remote peer 3256f4d6be2341febc82fc2eaaa6aff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:23.432855 2884 raft_consensus.cc:1261] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a [term 1 FOLLOWER]: Refusing update from remote peer 3256f4d6be2341febc82fc2eaaa6aff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 14:43:23.432998 2931 tablet.cc:2369] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:43:23.433599 2953 consensus_queue.cc:1035] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "372e62af0e6546ee97a11facfe56201c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 37127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:23.433799 2958 consensus_queue.cc:1035] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e1623375e1346c5ae95c825ff94401a" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:23.441645 2962 mvcc.cc:205] Tried to move back new op lower bound from 7333616858853482496 to 7333616858706620416. Current Snapshot: MvccSnapshot[applied={T|T < 7333616858853482496}]
I20260926 14:43:23.449095 2467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:43:23.449242 2467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:43:23.451362 2467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2226052e5c4f4aed97f1241eb8114d84: Sending DeleteTablet for 3 replicas of tablet 82535263cd81422294d0f8d739221d89
I20260926 14:43:23.452127 2595 tablet_service.cc:1514] Processing DeleteTablet for tablet 82535263cd81422294d0f8d739221d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:23 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:54802
I20260926 14:43:23.452250 2864 tablet_service.cc:1514] Processing DeleteTablet for tablet 82535263cd81422294d0f8d739221d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:23 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:47972
I20260926 14:43:23.452615 2730 tablet_service.cc:1514] Processing DeleteTablet for tablet 82535263cd81422294d0f8d739221d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:23 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:55978
I20260926 14:43:23.452682 2971 tablet_replica.cc:330] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6: stopping tablet replica
I20260926 14:43:23.452953 2971 raft_consensus.cc:2229] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:43:23.453161 2973 tablet_replica.cc:330] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c: stopping tablet replica
I20260926 14:43:23.453356 2971 raft_consensus.cc:2258] T 82535263cd81422294d0f8d739221d89 P 3256f4d6be2341febc82fc2eaaa6aff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:43:23.453560 2973 raft_consensus.cc:2229] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:43:23.453743 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 2531
I20260926 14:43:23.453944 2973 raft_consensus.cc:2258] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:43:23.455200 2973 ts_tablet_manager.cc:1905] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:43:23.457336 2973 ts_tablet_manager.cc:1918] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:43:23.457422 2973 log.cc:1196] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/82535263cd81422294d0f8d739221d89
I20260926 14:43:23.457648 2972 tablet_replica.cc:330] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a: stopping tablet replica
I20260926 14:43:23.457742 2973 ts_tablet_manager.cc:1939] T 82535263cd81422294d0f8d739221d89 P 372e62af0e6546ee97a11facfe56201c: Deleting consensus metadata
I20260926 14:43:23.457959 2972 raft_consensus.cc:2229] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:43:23.458271 2972 raft_consensus.cc:2258] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:43:23.458990 2454 catalog_manager.cc:4841] TS 372e62af0e6546ee97a11facfe56201c (127.16.90.194:37127): tablet 82535263cd81422294d0f8d739221d89 (table test-table [id=dc04574f26404c66b41f7c9ca3301fa8]) successfully deleted
I20260926 14:43:23.459641 2972 ts_tablet_manager.cc:1905] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 14:43:23.461160 2453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:42913 (error 108)
I20260926 14:43:23.461764 2972 ts_tablet_manager.cc:1918] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:43:23.461856 2972 log.cc:1196] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/82535263cd81422294d0f8d739221d89
I20260926 14:43:23.462018 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 2666
I20260926 14:43:23.462167 2972 ts_tablet_manager.cc:1939] T 82535263cd81422294d0f8d739221d89 P 4e1623375e1346c5ae95c825ff94401a: Deleting consensus metadata
I20260926 14:43:23.462234 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.461939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:42913 (local address 127.0.0.1:54814)
0926 14:43:23.462066 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:23.462144 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:42913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:43:23.462875 2453 catalog_manager.cc:4584] TS 3256f4d6be2341febc82fc2eaaa6aff6 (127.16.90.193:42913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82535263cd81422294d0f8d739221d89: Network error: Client connection negotiation failed: client connection to 127.16.90.193:42913: connect: Connection refused (error 111)
I20260926 14:43:23.464277 2453 catalog_manager.cc:4841] TS 4e1623375e1346c5ae95c825ff94401a (127.16.90.195:44515): tablet 82535263cd81422294d0f8d739221d89 (table test-table [id=dc04574f26404c66b41f7c9ca3301fa8]) successfully deleted
I20260926 14:43:23.468313 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 2801
I20260926 14:43:23.472971 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 2437
2026-09-26T14:43:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2215 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T14:43:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:23Z Disabled control of system clock
I20260926 14:43:23.501008 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:44831
--webserver_interface=127.16.90.254
--webserver_port=0
--builtin_ntp_servers=127.16.90.212:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:43:23.607772 2978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:23.608170 2978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:23.608237 2978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:23.614089 2978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:23.614153 2978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:23.614171 2978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:23.614188 2978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:23.614204 2978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:23.620791 2978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:44831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:23.621919 2978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:23.623296 2978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:23.631290 2984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:23.631311 2987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:23.631265 2983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:23.631605 2978 server_base.cc:1034] running on GCE node
I20260926 14:43:23.632149 2978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:23.633149 2978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:23.634313 2978 hybrid_clock.cc:648] HybridClock initialized: now 1790433803634290 us; error 37 us; skew 500 ppm
I20260926 14:43:23.636116 2978 webserver.cc:460] Webserver started at http://127.16.90.254:35569/ using document root <none> and password file <none>
I20260926 14:43:23.636655 2978 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:23.636721 2978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:23.636906 2978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:23.638882 2978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "b3fba6ea290d4a1d87c81cdc82b945f4"
format_stamp: "Formatted at 2026-09-26 14:43:23 on dist-test-slave-x4sn"
server_key: "883b4a72f43579599be491415b6cd278"
server_key_iv: "29cf902011168f9b4b75a4b4f0b90b33"
server_key_version: "encryptionkey@0"
I20260926 14:43:23.639403 2978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "b3fba6ea290d4a1d87c81cdc82b945f4"
format_stamp: "Formatted at 2026-09-26 14:43:23 on dist-test-slave-x4sn"
server_key: "883b4a72f43579599be491415b6cd278"
server_key_iv: "29cf902011168f9b4b75a4b4f0b90b33"
server_key_version: "encryptionkey@0"
I20260926 14:43:23.642866 2978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:43:23.645166 2993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:23.646346 2978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:43:23.646448 2978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "b3fba6ea290d4a1d87c81cdc82b945f4"
format_stamp: "Formatted at 2026-09-26 14:43:23 on dist-test-slave-x4sn"
server_key: "883b4a72f43579599be491415b6cd278"
server_key_iv: "29cf902011168f9b4b75a4b4f0b90b33"
server_key_version: "encryptionkey@0"
I20260926 14:43:23.646534 2978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:23.667176 2978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:23.667902 2978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:23.668068 2978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:23.674399 2978 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:44831
I20260926 14:43:23.674420 3045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:44831 every 8 connection(s)
I20260926 14:43:23.675302 2978 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:23.676828 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 2978
I20260926 14:43:23.676937 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:23.677212 16747 external_mini_cluster.cc:1428] Setting key a2116058de1f5373b1cebb6b7146f852
I20260926 14:43:23.679252 3046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:23.684764 3046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4: Bootstrap starting.
I20260926 14:43:23.685949 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.678429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.16.90.254:44831)
0926 14:43:23.679031 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:23.679046 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:23.679081 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:43:23.679794 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:23.679813 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:23.680201 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:23.680503 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:23.680924 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.681895 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.682752 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.683326 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.683469 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.684275 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:23.684280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:23.684293 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:23.684670 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:23.684724 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:23.684742 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:23.684824 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:23.685028 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 14:43:23.685266 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":126,"threads_started":1}
I20260926 14:43:23.686952 3046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:23.687654 3046 log.cc:824] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:23.689412 3046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4: No bootstrap required, opened a new log
I20260926 14:43:23.692679 3046 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fba6ea290d4a1d87c81cdc82b945f4" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44831 } }
I20260926 14:43:23.692889 3046 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:23.692943 3046 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fba6ea290d4a1d87c81cdc82b945f4, State: Initialized, Role: FOLLOWER
I20260926 14:43:23.693346 3046 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fba6ea290d4a1d87c81cdc82b945f4" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44831 } }
I20260926 14:43:23.693472 3046 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:23.693534 3046 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:23.693626 3046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:23.694432 3046 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fba6ea290d4a1d87c81cdc82b945f4" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44831 } }
I20260926 14:43:23.694727 3046 leader_election.cc:302] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3fba6ea290d4a1d87c81cdc82b945f4; no voters:
I20260926 14:43:23.695000 3046 leader_election.cc:288] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:23.695096 3051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:23.695308 3051 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [term 1 LEADER]: Becoming Leader. State: Replica: b3fba6ea290d4a1d87c81cdc82b945f4, State: Running, Role: LEADER
I20260926 14:43:23.695595 3051 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fba6ea290d4a1d87c81cdc82b945f4" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44831 } }
I20260926 14:43:23.696146 3046 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:23.697110 3053 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3fba6ea290d4a1d87c81cdc82b945f4. Latest consensus state: current_term: 1 leader_uuid: "b3fba6ea290d4a1d87c81cdc82b945f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fba6ea290d4a1d87c81cdc82b945f4" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44831 } } }
I20260926 14:43:23.697228 3053 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:23.697520 3052 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3fba6ea290d4a1d87c81cdc82b945f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fba6ea290d4a1d87c81cdc82b945f4" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 44831 } } }
I20260926 14:43:23.697626 3052 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:23.697609 3060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:23.700696 3060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:23.705811 3060 catalog_manager.cc:1347] Generated new cluster ID: fd69becc2ebe47f3be269460b0556169
I20260926 14:43:23.705870 3060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:23.718713 3060 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:23.719966 3060 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:23.726706 3060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4: Generated new TSK 0
I20260926 14:43:23.727463 3060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:23.731812 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44831
--builtin_ntp_servers=127.16.90.212:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:43:23.839447 3070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:23.839715 3070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:23.839764 3070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:23.843598 3070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:23.843648 3070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:23.843741 3070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:43:23.848325 3070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:23.849296 3070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:23.850911 3070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:23.859721 3076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:23.859762 3078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:23.859759 3075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:23.859875 3070 server_base.cc:1034] running on GCE node
I20260926 14:43:23.860270 3070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:23.860817 3070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:23.861963 3070 hybrid_clock.cc:648] HybridClock initialized: now 1790433803861942 us; error 37 us; skew 500 ppm
I20260926 14:43:23.863657 3070 webserver.cc:460] Webserver started at http://127.16.90.193:46615/ using document root <none> and password file <none>
I20260926 14:43:23.864209 3070 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:23.864269 3070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:23.864456 3070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:23.866335 3070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "c9a029e666674d1f945d747048c4d55a"
format_stamp: "Formatted at 2026-09-26 14:43:23 on dist-test-slave-x4sn"
server_key: "83f6f2c2e1a1c4f3d4de42c902359d7d"
server_key_iv: "0e23221203cd594fdeaf273a061cfb5f"
server_key_version: "encryptionkey@0"
I20260926 14:43:23.866832 3070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "c9a029e666674d1f945d747048c4d55a"
format_stamp: "Formatted at 2026-09-26 14:43:23 on dist-test-slave-x4sn"
server_key: "83f6f2c2e1a1c4f3d4de42c902359d7d"
server_key_iv: "0e23221203cd594fdeaf273a061cfb5f"
server_key_version: "encryptionkey@0"
I20260926 14:43:23.870193 3070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260926 14:43:23.872383 3084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:23.873482 3070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:43:23.873569 3070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "c9a029e666674d1f945d747048c4d55a"
format_stamp: "Formatted at 2026-09-26 14:43:23 on dist-test-slave-x4sn"
server_key: "83f6f2c2e1a1c4f3d4de42c902359d7d"
server_key_iv: "0e23221203cd594fdeaf273a061cfb5f"
server_key_version: "encryptionkey@0"
I20260926 14:43:23.873668 3070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:23.904076 3070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:23.904769 3070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:23.904943 3070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:23.905536 3070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:23.906450 3070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:23.906494 3070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:23.906531 3070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:23.906545 3070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:23.915107 3070 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:37809
I20260926 14:43:23.915143 3197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:37809 every 8 connection(s)
I20260926 14:43:23.916007 3070 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:43:23.917531 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 3070
I20260926 14:43:23.917650 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:43:23.917989 16747 external_mini_cluster.cc:1428] Setting key a9dcd8e8cb8beed9fef468e3281fb757
I20260926 14:43:23.920213 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44831
--builtin_ntp_servers=127.16.90.212:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:43:23.925750 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.917828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:40761 (local address 127.16.90.254:44831)
0926 14:43:23.918134 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:23.918140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:23.919165 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 14:43:23.920932 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:23.920935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:23.920984 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:23.921043 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:23.922116 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.922652 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.923853 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.924074 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.924162 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.925264 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:23.925268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:23.925270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:23.925284 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:23.925304 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:23.925311 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:23.925368 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:23.925517 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 14:43:23.925602 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260926 14:43:23.926577 3200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:23.918286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44831 (local address 127.16.90.193:40761)
0926 14:43:23.919029 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:23.919071 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:23.920695 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:23.921237 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:23.921246 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:23.921354 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:23.921972 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.921986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.922774 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.922777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.923736 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:23.923747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:23.924213 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:23.924218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:23.924384 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:23.924978 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:43:23.924980 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:23.925059 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:43:23.925124 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:23.925376 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:23.925386 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:23.925391 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:43:23.925499 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 14:43:23.925775 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":147,"threads_started":1}
I20260926 14:43:23.927918 3198 heartbeater.cc:347] Connected to a master server at 127.16.90.254:44831
I20260926 14:43:23.928169 3198 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:23.928659 3198 heartbeater.cc:511] Master 127.16.90.254:44831 requested a full tablet report, sending...
I20260926 14:43:23.930274 3010 ts_manager.cc:194] Registered new tserver with Master: c9a029e666674d1f945d747048c4d55a (127.16.90.193:37809)
I20260926 14:43:23.931844 3010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.90.193:40761
W20260926 14:43:24.030891 3201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:24.031149 3201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:24.031185 3201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:24.035105 3201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:24.035159 3201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:24.035212 3201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:43:24.039901 3201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:24.040922 3201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:24.042274 3201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:24.051331 3207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:24.051342 3209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:24.051398 3201 server_base.cc:1034] running on GCE node
W20260926 14:43:24.051342 3206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:24.052027 3201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:24.052631 3201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:24.053789 3201 hybrid_clock.cc:648] HybridClock initialized: now 1790433804053775 us; error 28 us; skew 500 ppm
I20260926 14:43:24.055558 3201 webserver.cc:460] Webserver started at http://127.16.90.194:45259/ using document root <none> and password file <none>
I20260926 14:43:24.056123 3201 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:24.056196 3201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:24.056381 3201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:24.058359 3201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "7cc3e4ec29df4058953b2cd604a9cf73"
format_stamp: "Formatted at 2026-09-26 14:43:24 on dist-test-slave-x4sn"
server_key: "899233042d7cb6c885e07b587b336c57"
server_key_iv: "209456f8cdc5ed2833fad81e2ae78250"
server_key_version: "encryptionkey@0"
I20260926 14:43:24.058881 3201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "7cc3e4ec29df4058953b2cd604a9cf73"
format_stamp: "Formatted at 2026-09-26 14:43:24 on dist-test-slave-x4sn"
server_key: "899233042d7cb6c885e07b587b336c57"
server_key_iv: "209456f8cdc5ed2833fad81e2ae78250"
server_key_version: "encryptionkey@0"
I20260926 14:43:24.062484 3201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:43:24.064872 3215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:24.066015 3201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:43:24.066118 3201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "7cc3e4ec29df4058953b2cd604a9cf73"
format_stamp: "Formatted at 2026-09-26 14:43:24 on dist-test-slave-x4sn"
server_key: "899233042d7cb6c885e07b587b336c57"
server_key_iv: "209456f8cdc5ed2833fad81e2ae78250"
server_key_version: "encryptionkey@0"
I20260926 14:43:24.066217 3201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:24.094906 3201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:24.095695 3201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:24.095866 3201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:24.096478 3201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:24.097544 3201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:24.097602 3201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:24.097659 3201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:24.097685 3201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:24.105798 3201 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:33721
I20260926 14:43:24.105832 3328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:33721 every 8 connection(s)
I20260926 14:43:24.106654 3201 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:43:24.106813 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 3201
I20260926 14:43:24.106922 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:43:24.107271 16747 external_mini_cluster.cc:1428] Setting key a3b8192e07569ce2afca51725119467d
I20260926 14:43:24.109406 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44831
--builtin_ntp_servers=127.16.90.212:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:43:24.116267 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.108434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:37475 (local address 127.16.90.254:44831)
0926 14:43:24.108562 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:24.108567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:24.109786 (+ 1219us) server_negotiation.cc:407] Connection header received
0926 14:43:24.111356 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:24.111359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:24.111409 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:24.111499 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:24.112469 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.113045 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.114276 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.114501 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.114592 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.115795 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:24.115799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:24.115801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:24.115813 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:24.115832 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:24.115840 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:24.115899 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:24.116067 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 14:43:24.116125 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:43:24.117178 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.108927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44831 (local address 127.16.90.194:37475)
0926 14:43:24.109647 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:24.109697 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:24.111100 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:24.111727 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:24.111737 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:24.111847 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:24.112333 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.112345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.113183 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.113187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.114149 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.114159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.114657 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.114662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.114822 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.115522 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:43:24.115523 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:24.115564 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:43:24.115627 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:24.115911 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:24.115920 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:24.115926 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:43:24.116032 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 14:43:24.116317 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":139,"threads_started":1}
I20260926 14:43:24.118431 3329 heartbeater.cc:347] Connected to a master server at 127.16.90.254:44831
I20260926 14:43:24.118690 3329 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:24.119272 3329 heartbeater.cc:511] Master 127.16.90.254:44831 requested a full tablet report, sending...
I20260926 14:43:24.120396 3010 ts_manager.cc:194] Registered new tserver with Master: 7cc3e4ec29df4058953b2cd604a9cf73 (127.16.90.194:33721)
I20260926 14:43:24.121099 3010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.90.194:37475
W20260926 14:43:24.220953 3332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:24.221225 3332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:24.221262 3332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:24.225260 3332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:24.225317 3332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:24.225371 3332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:43:24.230104 3332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:24.231083 3332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:24.232393 3332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:24.241252 3338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:24.241345 3337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:24.241286 3340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:24.241544 3332 server_base.cc:1034] running on GCE node
I20260926 14:43:24.241925 3332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:24.242537 3332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:24.243713 3332 hybrid_clock.cc:648] HybridClock initialized: now 1790433804243683 us; error 37 us; skew 500 ppm
I20260926 14:43:24.245386 3332 webserver.cc:460] Webserver started at http://127.16.90.195:45601/ using document root <none> and password file <none>
I20260926 14:43:24.245937 3332 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:24.245996 3332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:24.246192 3332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:24.248317 3332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "12d7cf97a4d74e459818d95f69b3ef55"
format_stamp: "Formatted at 2026-09-26 14:43:24 on dist-test-slave-x4sn"
server_key: "69bc855fd6e5f99d7667c8ed6113f473"
server_key_iv: "0027b29b9d97a3808e9f8bbefd5c37c5"
server_key_version: "encryptionkey@0"
I20260926 14:43:24.248831 3332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "12d7cf97a4d74e459818d95f69b3ef55"
format_stamp: "Formatted at 2026-09-26 14:43:24 on dist-test-slave-x4sn"
server_key: "69bc855fd6e5f99d7667c8ed6113f473"
server_key_iv: "0027b29b9d97a3808e9f8bbefd5c37c5"
server_key_version: "encryptionkey@0"
I20260926 14:43:24.252373 3332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:24.254539 3346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:24.255726 3332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:24.255821 3332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "12d7cf97a4d74e459818d95f69b3ef55"
format_stamp: "Formatted at 2026-09-26 14:43:24 on dist-test-slave-x4sn"
server_key: "69bc855fd6e5f99d7667c8ed6113f473"
server_key_iv: "0027b29b9d97a3808e9f8bbefd5c37c5"
server_key_version: "encryptionkey@0"
I20260926 14:43:24.255904 3332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:24.279808 3332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:24.280472 3332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:24.280637 3332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:24.281208 3332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:24.282122 3332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:24.282176 3332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:24.282243 3332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:24.282266 3332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:24.290390 3332 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:35521
I20260926 14:43:24.290439 3459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:35521 every 8 connection(s)
I20260926 14:43:24.291232 3332 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:43:24.296329 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 3332
I20260926 14:43:24.296429 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:43:24.296707 16747 external_mini_cluster.cc:1428] Setting key 4396af75fccfd3b75c4de2c74b39de59
I20260926 14:43:24.299968 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.292921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:45333 (local address 127.16.90.254:44831)
0926 14:43:24.293049 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:24.293053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:24.294146 (+ 1093us) server_negotiation.cc:407] Connection header received
0926 14:43:24.295459 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:24.295462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:24.295508 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:24.295588 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:24.296475 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.297099 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.298215 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.298443 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.298529 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.299504 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:24.299507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:24.299510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:24.299522 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:24.299542 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:24.299549 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:24.299605 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:24.299776 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 14:43:24.299833 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:43:24.300726 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.293340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:44831 (local address 127.16.90.195:45333)
0926 14:43:24.293974 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:24.294013 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:24.295260 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:24.295770 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:24.295777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:24.295859 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:24.296341 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.296353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.297209 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.297213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.298117 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.298127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.298569 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.298573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.298736 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.299272 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:43:24.299273 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:24.299311 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:43:24.299369 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:24.299614 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:24.299622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:24.299628 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:43:24.299768 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 14:43:24.300021 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":108,"threads_started":1}
I20260926 14:43:24.301631 3460 heartbeater.cc:347] Connected to a master server at 127.16.90.254:44831
I20260926 14:43:24.301836 3460 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:24.302284 3460 heartbeater.cc:511] Master 127.16.90.254:44831 requested a full tablet report, sending...
I20260926 14:43:24.303141 3010 ts_manager.cc:194] Registered new tserver with Master: 12d7cf97a4d74e459818d95f69b3ef55 (127.16.90.195:35521)
I20260926 14:43:24.303647 3010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.90.195:45333
I20260926 14:43:24.310015 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:43:24.314942 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.311777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.16.90.254:44831)
0926 14:43:24.311894 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:24.311898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:24.312053 (+ 155us) server_negotiation.cc:407] Connection header received
0926 14:43:24.312141 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:24.312143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:24.312200 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:24.312257 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:24.312461 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.312934 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.313688 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.313884 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.313949 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.314578 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:24.314581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:24.314583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:24.314593 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:24.314610 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:24.314618 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:24.314710 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:24.314787 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 14:43:24.314814 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:43:24.317329 3010 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:43:24.320214 3010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:43:24.333240 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.327718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:33721 (local address 127.0.0.1:37392)
0926 14:43:24.328448 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:24.328465 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:24.328564 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:24.329397 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:24.329403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:24.329422 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:24.329553 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.329560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.330562 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.330565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.331760 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.331770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.332712 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.332715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.332827 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.332862 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 14:43:24.332970 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 14:43:24.333084 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":22,"thread_start_us":68,"threads_started":1}
I20260926 14:43:24.333246 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.327856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.16.90.194:33721)
0926 14:43:24.328904 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:24.328917 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:24.328939 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:43:24.329015 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:24.329020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:24.329247 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:24.329398 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:24.329661 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.330459 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.331890 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.332609 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.332709 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.332899 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:24.332998 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 14:43:24.333091 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":103,"threads_started":1}
I20260926 14:43:24.333796 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.328072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:37809 (local address 127.0.0.1:60514)
0926 14:43:24.328822 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:24.328834 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:24.328919 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:24.329555 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:24.329557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:24.329574 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:24.329665 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.329671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.331075 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.331079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.332317 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.332326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.333457 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.333461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.333558 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.333576 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:43:24.333633 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:43:24.333691 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":73,"thread_start_us":95,"threads_started":1}
I20260926 14:43:24.333945 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.328467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.16.90.193:37809)
0926 14:43:24.328860 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:24.328872 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:24.328899 (+ 27us) server_negotiation.cc:407] Connection header received
0926 14:43:24.329073 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:24.329078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:24.329272 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:24.329414 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:24.329776 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.330948 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.332439 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.333167 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.333253 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.333400 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:24.333697 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 14:43:24.333796 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":91,"threads_started":1}
I20260926 14:43:24.335039 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.327816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:35521 (local address 127.0.0.1:56056)
0926 14:43:24.328356 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:24.328414 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:24.328527 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:24.329681 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:24.329683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:24.329702 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:24.329783 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.329788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.331099 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.331102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.332189 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.332198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.333457 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.333461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.334732 (+ 1271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.334768 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 14:43:24.334837 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:43:24.334900 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":555,"thread_start_us":75,"threads_started":1}
I20260926 14:43:24.335505 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.328465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.16.90.195:35521)
0926 14:43:24.329109 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:24.329120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:24.329139 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:43:24.329198 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:24.329202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:24.329446 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:24.329583 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:24.329893 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.330977 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.332294 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.333049 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.333149 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.333506 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:24.335261 (+ 1755us) server_negotiation.cc:299] Negotiation successful
0926 14:43:24.335358 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":122,"threads_started":1}
I20260926 14:43:24.336505 3263 tablet_service.cc:1467] Processing CreateTablet for tablet 1f023a2ef73f42d591b1805576fb7882 (DEFAULT_TABLE table=test-table [id=d4ed830a93ff4d7598b781e487876b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:24.336515 3132 tablet_service.cc:1467] Processing CreateTablet for tablet 1f023a2ef73f42d591b1805576fb7882 (DEFAULT_TABLE table=test-table [id=d4ed830a93ff4d7598b781e487876b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:24.337427 3263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f023a2ef73f42d591b1805576fb7882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:24.337532 3132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f023a2ef73f42d591b1805576fb7882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:24.337987 3394 tablet_service.cc:1467] Processing CreateTablet for tablet 1f023a2ef73f42d591b1805576fb7882 (DEFAULT_TABLE table=test-table [id=d4ed830a93ff4d7598b781e487876b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:24.338871 3394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f023a2ef73f42d591b1805576fb7882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:24.342931 3477 tablet_bootstrap.cc:492] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73: Bootstrap starting.
I20260926 14:43:24.344883 3477 tablet_bootstrap.cc:654] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:24.345099 3478 tablet_bootstrap.cc:492] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55: Bootstrap starting.
I20260926 14:43:24.346218 3477 log.cc:824] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:24.346886 3479 tablet_bootstrap.cc:492] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a: Bootstrap starting.
I20260926 14:43:24.347149 3478 tablet_bootstrap.cc:654] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:24.348106 3477 tablet_bootstrap.cc:492] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73: No bootstrap required, opened a new log
I20260926 14:43:24.348301 3477 ts_tablet_manager.cc:1397] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:43:24.348345 3478 log.cc:824] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:24.348884 3479 tablet_bootstrap.cc:654] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:24.349630 3479 log.cc:824] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:24.350263 3478 tablet_bootstrap.cc:492] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55: No bootstrap required, opened a new log
I20260926 14:43:24.350426 3478 ts_tablet_manager.cc:1397] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:43:24.351543 3479 tablet_bootstrap.cc:492] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a: No bootstrap required, opened a new log
I20260926 14:43:24.351738 3479 ts_tablet_manager.cc:1397] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260926 14:43:24.351845 3477 raft_consensus.cc:348] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc3e4ec29df4058953b2cd604a9cf73" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33721 } } peers { permanent_uuid: "12d7cf97a4d74e459818d95f69b3ef55" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 35521 } } peers { permanent_uuid: "c9a029e666674d1f945d747048c4d55a" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37809 } }
I20260926 14:43:24.352064 3477 raft_consensus.cc:374] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:24.352113 3477 raft_consensus.cc:729] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc3e4ec29df4058953b2cd604a9cf73, State: Initialized, Role: FOLLOWER
I20260926 14:43:24.352523 3477 consensus_queue.cc:260] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc3e4ec29df4058953b2cd604a9cf73" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33721 } } peers { permanent_uuid: "12d7cf97a4d74e459818d95f69b3ef55" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 35521 } } peers { permanent_uuid: "c9a029e666674d1f945d747048c4d55a" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37809 } }
I20260926 14:43:24.353421 3477 ts_tablet_manager.cc:1428] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:43:24.353732 3329 heartbeater.cc:503] Master 127.16.90.254:44831 was elected leader, sending a full tablet report...
I20260926 14:43:24.354667 3478 raft_consensus.cc:348] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc3e4ec29df4058953b2cd604a9cf73" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33721 } } peers { permanent_uuid: "12d7cf97a4d74e459818d95f69b3ef55" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 35521 } } peers { permanent_uuid: "c9a029e666674d1f945d747048c4d55a" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37809 } }
I20260926 14:43:24.354907 3478 raft_consensus.cc:374] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:24.354957 3478 raft_consensus.cc:729] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d7cf97a4d74e459818d95f69b3ef55, State: Initialized, Role: FOLLOWER
I20260926 14:43:24.355355 3478 consensus_queue.cc:260] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc3e4ec29df4058953b2cd604a9cf73" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33721 } } peers { permanent_uuid: "12d7cf97a4d74e459818d95f69b3ef55" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 35521 } } peers { permanent_uuid: "c9a029e666674d1f945d747048c4d55a" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37809 } }
I20260926 14:43:24.355640 3479 raft_consensus.cc:348] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc3e4ec29df4058953b2cd604a9cf73" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33721 } } peers { permanent_uuid: "12d7cf97a4d74e459818d95f69b3ef55" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 35521 } } peers { permanent_uuid: "c9a029e666674d1f945d747048c4d55a" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37809 } }
I20260926 14:43:24.355899 3479 raft_consensus.cc:374] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:24.355957 3479 raft_consensus.cc:729] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a029e666674d1f945d747048c4d55a, State: Initialized, Role: FOLLOWER
I20260926 14:43:24.356079 3460 heartbeater.cc:503] Master 127.16.90.254:44831 was elected leader, sending a full tablet report...
I20260926 14:43:24.356297 3478 ts_tablet_manager.cc:1428] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:43:24.356381 3479 consensus_queue.cc:260] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc3e4ec29df4058953b2cd604a9cf73" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33721 } } peers { permanent_uuid: "12d7cf97a4d74e459818d95f69b3ef55" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 35521 } } peers { permanent_uuid: "c9a029e666674d1f945d747048c4d55a" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37809 } }
I20260926 14:43:24.356937 3198 heartbeater.cc:503] Master 127.16.90.254:44831 was elected leader, sending a full tablet report...
I20260926 14:43:24.357406 3479 ts_tablet_manager.cc:1428] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260926 14:43:24.357579 3330 tablet.cc:2369] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:43:24.417115 3199 tablet.cc:2369] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:43:24.478932 3484 raft_consensus.cc:482] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:43:24.479148 3484 raft_consensus.cc:504] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc3e4ec29df4058953b2cd604a9cf73" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33721 } } peers { permanent_uuid: "12d7cf97a4d74e459818d95f69b3ef55" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 35521 } } peers { permanent_uuid: "c9a029e666674d1f945d747048c4d55a" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37809 } }
I20260926 14:43:24.480355 3484 leader_election.cc:288] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cc3e4ec29df4058953b2cd604a9cf73 (127.16.90.194:33721), c9a029e666674d1f945d747048c4d55a (127.16.90.193:37809)
I20260926 14:43:24.483879 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.480575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:33721 (local address 127.16.90.195:57469)
0926 14:43:24.480720 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:24.480735 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:24.480830 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:24.481108 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:24.481111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:24.481127 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:24.481252 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.481260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.482024 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.482027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.482835 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.482844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.483480 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.483484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.483573 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.483594 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:43:24.483696 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:43:24.483766 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 14:43:24.483958 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.480667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:57469 (local address 127.16.90.194:33721)
0926 14:43:24.480792 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:24.480796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:24.480811 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:43:24.480946 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:24.480950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:24.481007 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:24.481092 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:24.481349 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.481924 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.482948 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.483372 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.483462 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.483523 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:24.483750 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 14:43:24.483834 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:43:24.484079 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.480602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:37809 (local address 127.16.90.195:41571)
0926 14:43:24.481005 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:24.481018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:24.481106 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:24.481339 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:24.481341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:24.481356 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:24.481473 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.481479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.482167 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.482170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.482911 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:24.482921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.483531 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.483534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:24.483750 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.483781 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 14:43:24.483857 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 14:43:24.483918 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":127,"threads_started":1}
I20260926 14:43:24.484081 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.480813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:41571 (local address 127.16.90.193:37809)
0926 14:43:24.480927 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:24.480931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:24.481081 (+ 150us) server_negotiation.cc:407] Connection header received
0926 14:43:24.481194 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:24.481197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:24.481243 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:24.481315 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:24.481565 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.482065 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.483057 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.483423 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.483528 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.483575 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:24.483900 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 14:43:24.483968 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:43:24.484755 3283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f023a2ef73f42d591b1805576fb7882" candidate_uuid: "12d7cf97a4d74e459818d95f69b3ef55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc3e4ec29df4058953b2cd604a9cf73" is_pre_election: true
I20260926 14:43:24.484720 3152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f023a2ef73f42d591b1805576fb7882" candidate_uuid: "12d7cf97a4d74e459818d95f69b3ef55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a029e666674d1f945d747048c4d55a" is_pre_election: true
I20260926 14:43:24.485049 3283 raft_consensus.cc:2454] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d7cf97a4d74e459818d95f69b3ef55 in term 0.
I20260926 14:43:24.485055 3152 raft_consensus.cc:2454] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d7cf97a4d74e459818d95f69b3ef55 in term 0.
I20260926 14:43:24.485572 3347 leader_election.cc:302] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d7cf97a4d74e459818d95f69b3ef55, 7cc3e4ec29df4058953b2cd604a9cf73; no voters:
I20260926 14:43:24.485849 3484 raft_consensus.cc:2781] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:43:24.485945 3484 raft_consensus.cc:482] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:43:24.485997 3484 raft_consensus.cc:3037] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:24.486976 3484 raft_consensus.cc:504] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc3e4ec29df4058953b2cd604a9cf73" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33721 } } peers { permanent_uuid: "12d7cf97a4d74e459818d95f69b3ef55" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 35521 } } peers { permanent_uuid: "c9a029e666674d1f945d747048c4d55a" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37809 } }
I20260926 14:43:24.487299 3484 leader_election.cc:288] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [CANDIDATE]: Term 1 election: Requested vote from peers 7cc3e4ec29df4058953b2cd604a9cf73 (127.16.90.194:33721), c9a029e666674d1f945d747048c4d55a (127.16.90.193:37809)
I20260926 14:43:24.487624 3283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f023a2ef73f42d591b1805576fb7882" candidate_uuid: "12d7cf97a4d74e459818d95f69b3ef55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc3e4ec29df4058953b2cd604a9cf73"
I20260926 14:43:24.487763 3283 raft_consensus.cc:3037] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:24.488724 3283 raft_consensus.cc:2454] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d7cf97a4d74e459818d95f69b3ef55 in term 1.
I20260926 14:43:24.488973 3152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f023a2ef73f42d591b1805576fb7882" candidate_uuid: "12d7cf97a4d74e459818d95f69b3ef55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a029e666674d1f945d747048c4d55a"
I20260926 14:43:24.489042 3347 leader_election.cc:302] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d7cf97a4d74e459818d95f69b3ef55, 7cc3e4ec29df4058953b2cd604a9cf73; no voters:
I20260926 14:43:24.489106 3152 raft_consensus.cc:3037] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:24.489188 3484 raft_consensus.cc:2781] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:24.489406 3484 raft_consensus.cc:686] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 1 LEADER]: Becoming Leader. State: Replica: 12d7cf97a4d74e459818d95f69b3ef55, State: Running, Role: LEADER
I20260926 14:43:24.489768 3484 consensus_queue.cc:237] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc3e4ec29df4058953b2cd604a9cf73" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33721 } } peers { permanent_uuid: "12d7cf97a4d74e459818d95f69b3ef55" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 35521 } } peers { permanent_uuid: "c9a029e666674d1f945d747048c4d55a" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37809 } }
I20260926 14:43:24.490155 3152 raft_consensus.cc:2454] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d7cf97a4d74e459818d95f69b3ef55 in term 1.
I20260926 14:43:24.492980 3009 catalog_manager.cc:5495] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12d7cf97a4d74e459818d95f69b3ef55 (127.16.90.195). New cstate: current_term: 1 leader_uuid: "12d7cf97a4d74e459818d95f69b3ef55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc3e4ec29df4058953b2cd604a9cf73" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d7cf97a4d74e459818d95f69b3ef55" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 35521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a029e666674d1f945d747048c4d55a" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37809 } health_report { overall_health: UNKNOWN } } }
I20260926 14:43:24.530959 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.527788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.16.90.195:35521)
0926 14:43:24.527916 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:24.527920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:24.527933 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:43:24.528007 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:24.528009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:24.528061 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:24.528148 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:24.528358 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.528836 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.529669 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:24.529867 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:24.529934 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:24.530178 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:24.530572 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:24.530681 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:24.530761 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 14:43:24.530795 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:43:24.536876 3283 raft_consensus.cc:1261] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73 [term 1 FOLLOWER]: Refusing update from remote peer 12d7cf97a4d74e459818d95f69b3ef55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:24.536885 3152 raft_consensus.cc:1261] T 1f023a2ef73f42d591b1805576fb7882 P c9a029e666674d1f945d747048c4d55a [term 1 FOLLOWER]: Refusing update from remote peer 12d7cf97a4d74e459818d95f69b3ef55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:24.537487 3484 consensus_queue.cc:1035] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a029e666674d1f945d747048c4d55a" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 37809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:24.537669 3487 consensus_queue.cc:1035] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cc3e4ec29df4058953b2cd604a9cf73" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 33721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:43:24.542100 3461 tablet.cc:2369] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:43:24.544889 3491 mvcc.cc:205] Tried to move back new op lower bound from 7333616863375335424 to 7333616863193153536. Current Snapshot: MvccSnapshot[applied={T|T < 7333616863375335424}]
I20260926 14:43:24.548352 3493 mvcc.cc:205] Tried to move back new op lower bound from 7333616863375335424 to 7333616863193153536. Current Snapshot: MvccSnapshot[applied={T|T < 7333616863375335424}]
I20260926 14:43:24.553869 3009 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:43:24.554009 3009 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:43:24.556125 3009 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3fba6ea290d4a1d87c81cdc82b945f4: Sending DeleteTablet for 3 replicas of tablet 1f023a2ef73f42d591b1805576fb7882
I20260926 14:43:24.556775 3394 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f023a2ef73f42d591b1805576fb7882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:24 UTC) from {username='slave'} at 127.0.0.1:56056
I20260926 14:43:24.556856 3132 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f023a2ef73f42d591b1805576fb7882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:24 UTC) from {username='slave'} at 127.0.0.1:60514
I20260926 14:43:24.556887 3263 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f023a2ef73f42d591b1805576fb7882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:24 UTC) from {username='slave'} at 127.0.0.1:37392
I20260926 14:43:24.557281 3501 tablet_replica.cc:330] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55: stopping tablet replica
I20260926 14:43:24.557580 3501 raft_consensus.cc:2229] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:43:24.557600 3502 tablet_replica.cc:330] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73: stopping tablet replica
I20260926 14:43:24.557689 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 3070
I20260926 14:43:24.557862 3502 raft_consensus.cc:2229] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:43:24.557920 3501 raft_consensus.cc:2258] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:43:24.558146 3502 raft_consensus.cc:2258] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:43:24.559417 3501 ts_tablet_manager.cc:1905] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:43:24.559504 3502 ts_tablet_manager.cc:1905] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:43:24.562022 3502 ts_tablet_manager.cc:1918] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:43:24.562112 3502 log.cc:1196] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/1f023a2ef73f42d591b1805576fb7882
I20260926 14:43:24.562402 3502 ts_tablet_manager.cc:1939] T 1f023a2ef73f42d591b1805576fb7882 P 7cc3e4ec29df4058953b2cd604a9cf73: Deleting consensus metadata
I20260926 14:43:24.562569 3501 ts_tablet_manager.cc:1918] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:43:24.562642 3501 log.cc:1196] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/1f023a2ef73f42d591b1805576fb7882
I20260926 14:43:24.562922 3501 ts_tablet_manager.cc:1939] T 1f023a2ef73f42d591b1805576fb7882 P 12d7cf97a4d74e459818d95f69b3ef55: Deleting consensus metadata
I20260926 14:43:24.563570 2994 catalog_manager.cc:4841] TS 7cc3e4ec29df4058953b2cd604a9cf73 (127.16.90.194:33721): tablet 1f023a2ef73f42d591b1805576fb7882 (table test-table [id=d4ed830a93ff4d7598b781e487876b48]) successfully deleted
I20260926 14:43:24.564007 2997 catalog_manager.cc:4841] TS 12d7cf97a4d74e459818d95f69b3ef55 (127.16.90.195:35521): tablet 1f023a2ef73f42d591b1805576fb7882 (table test-table [id=d4ed830a93ff4d7598b781e487876b48]) successfully deleted
W20260926 14:43:24.567894 2995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:37809 (error 108)
I20260926 14:43:24.568672 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 3201
I20260926 14:43:24.568846 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:24.568573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:37809 (local address 127.0.0.1:60526)
0926 14:43:24.568688 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:24.568760 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:37809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 14:43:24.569831 2995 catalog_manager.cc:4584] TS c9a029e666674d1f945d747048c4d55a (127.16.90.193:37809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f023a2ef73f42d591b1805576fb7882: Network error: Client connection negotiation failed: client connection to 127.16.90.193:37809: connect: Connection refused (error 111)
I20260926 14:43:24.573829 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 3332
I20260926 14:43:24.578462 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 2978
2026-09-26T14:43:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1104 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:24 dist-test-slave-x4sn krb5kdc[3507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:24 dist-test-slave-x4sn krb5kdc[3507](info): set up 2 sockets
Sep 26 14:43:24 dist-test-slave-x4sn krb5kdc[3507](info): commencing operation
krb5kdc: starting...
W20260926 14:43:25.525024 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:25 dist-test-slave-x4sn krb5kdc[3507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:43:25.640581 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:36577
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:36577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:25.746989 3520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:25.747257 3520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:25.747313 3520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:25.751250 3520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:25.751307 3520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:25.751332 3520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:25.751353 3520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:25.751377 3520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:25.756487 3520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:36577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:36577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:25.757483 3520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:25.758456 3520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:25.766304 3525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:25.766367 3526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:25.766426 3520 server_base.cc:1034] running on GCE node
W20260926 14:43:25.766626 3528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:25.767143 3520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:25.768244 3520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:25.769398 3520 hybrid_clock.cc:648] HybridClock initialized: now 1790433805769380 us; error 31 us; skew 500 ppm
Sep 26 14:43:25 dist-test-slave-x4sn krb5kdc[3507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433805, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:25.771917 3520 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:43:25.772753 3520 webserver.cc:460] Webserver started at http://127.16.90.254:33001/ using document root <none> and password file <none>
I20260926 14:43:25.773275 3520 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:25.773340 3520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:25.773535 3520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:25.775490 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "1431d46499fd45cc988e338bba1e8072"
format_stamp: "Formatted at 2026-09-26 14:43:25 on dist-test-slave-x4sn"
server_key: "a92635cbe8cc034a2a0b7cbce2065b0d"
server_key_iv: "8aaed74215251b269c14ad6b929919da"
server_key_version: "encryptionkey@0"
I20260926 14:43:25.776075 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "1431d46499fd45cc988e338bba1e8072"
format_stamp: "Formatted at 2026-09-26 14:43:25 on dist-test-slave-x4sn"
server_key: "a92635cbe8cc034a2a0b7cbce2065b0d"
server_key_iv: "8aaed74215251b269c14ad6b929919da"
server_key_version: "encryptionkey@0"
I20260926 14:43:25.779520 3520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:25.781791 3535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:25.782945 3520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:25.783054 3520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "1431d46499fd45cc988e338bba1e8072"
format_stamp: "Formatted at 2026-09-26 14:43:25 on dist-test-slave-x4sn"
server_key: "a92635cbe8cc034a2a0b7cbce2065b0d"
server_key_iv: "8aaed74215251b269c14ad6b929919da"
server_key_version: "encryptionkey@0"
I20260926 14:43:25.783145 3520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:25.823741 3520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:25.826639 3520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:25.826817 3520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:25.833372 3520 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:36577
I20260926 14:43:25.833388 3587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:36577 every 8 connection(s)
I20260926 14:43:25.834280 3520 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:25.836539 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 3520
I20260926 14:43:25.836643 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:25.836925 16747 external_mini_cluster.cc:1428] Setting key 830c1fe1c2e6296000215696c82c7127
I20260926 14:43:25.836938 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:25.843878 3588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072: Bootstrap starting.
Sep 26 14:43:25 dist-test-slave-x4sn krb5kdc[3507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:25.846199 3588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:25.847012 3588 log.cc:824] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:25.849128 3588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072: No bootstrap required, opened a new log
I20260926 14:43:25.850876 3591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:25.838240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.16.90.254:36577)
0926 14:43:25.838644 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:25.838662 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:25.838704 (+ 42us) server_negotiation.cc:407] Connection header received
0926 14:43:25.839538 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:25.839561 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:25.839997 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:25.840376 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:25.840971 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:25.842437 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:25.843221 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:25.843911 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:25.844065 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:25.846522 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:25.846527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:25.846542 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:25.846572 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:25.848228 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:25.848667 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:25.848671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:25.848675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:25.848775 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:25.849056 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:25.849058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:25.849059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:25.849465 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:25.849601 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:25.849889 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 14:43:25.850164 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260926 14:43:25.852981 3588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1431d46499fd45cc988e338bba1e8072" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36577 } }
I20260926 14:43:25.853201 3588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:25.853259 3588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1431d46499fd45cc988e338bba1e8072, State: Initialized, Role: FOLLOWER
I20260926 14:43:25.853645 3588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1431d46499fd45cc988e338bba1e8072" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36577 } }
I20260926 14:43:25.853777 3588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:25.853839 3588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:25.853935 3588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:25.854918 3588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1431d46499fd45cc988e338bba1e8072" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36577 } }
I20260926 14:43:25.855226 3588 leader_election.cc:302] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1431d46499fd45cc988e338bba1e8072; no voters:
I20260926 14:43:25.855504 3588 leader_election.cc:288] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:25.855595 3593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:25.855808 3593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [term 1 LEADER]: Becoming Leader. State: Replica: 1431d46499fd45cc988e338bba1e8072, State: Running, Role: LEADER
I20260926 14:43:25.856094 3593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1431d46499fd45cc988e338bba1e8072" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36577 } }
I20260926 14:43:25.856647 3588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:25.857642 3594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1431d46499fd45cc988e338bba1e8072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1431d46499fd45cc988e338bba1e8072" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36577 } } }
I20260926 14:43:25.857671 3595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1431d46499fd45cc988e338bba1e8072. Latest consensus state: current_term: 1 leader_uuid: "1431d46499fd45cc988e338bba1e8072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1431d46499fd45cc988e338bba1e8072" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 36577 } } }
I20260926 14:43:25.857795 3595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:25.857795 3594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:25.858244 3602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:25.861143 3602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:25.866727 3602 catalog_manager.cc:1347] Generated new cluster ID: 2b66acad8a204c0c9cb643326c8c5bde
I20260926 14:43:25.866789 3602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:25.895030 3602 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:25.896359 3602 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:25.902864 3602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072: Generated new TSK 0
I20260926 14:43:25.903643 3602 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.90.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
Entry for principal kudu/127.16.90.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab.
I20260926 14:43:25.931833 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:36577
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:26.041815 3614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:26.042090 3614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:26.042150 3614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:26.046121 3614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:26.046185 3614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:26.046262 3614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:43:26.051498 3614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.193.keytab
--principal=kudu/127.16.90.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:26.052613 3614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:26.053619 3614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:26.062610 3622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:26.062603 3619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:26.062718 3614 server_base.cc:1034] running on GCE node
W20260926 14:43:26.062598 3620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:26.063292 3614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:26.063916 3614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:26.065085 3614 hybrid_clock.cc:648] HybridClock initialized: now 1790433806065066 us; error 30 us; skew 500 ppm
Sep 26 14:43:26 dist-test-slave-x4sn krb5kdc[3507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:26.067596 3614 init.cc:377] Logged in from keytab as kudu/127.16.90.193@KRBTEST.COM (short username kudu)
I20260926 14:43:26.068395 3614 webserver.cc:460] Webserver started at http://127.16.90.193:35923/ using document root <none> and password file <none>
I20260926 14:43:26.068907 3614 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:26.068971 3614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:26.069165 3614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:26.071071 3614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "5015ef0767ae42858a246b45f6e20395"
format_stamp: "Formatted at 2026-09-26 14:43:26 on dist-test-slave-x4sn"
server_key: "8fb13c4a28ee0c1649b84019d26c1d96"
server_key_iv: "27e057e4b94218d6f6d8c089ed1e2385"
server_key_version: "encryptionkey@0"
I20260926 14:43:26.071573 3614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "5015ef0767ae42858a246b45f6e20395"
format_stamp: "Formatted at 2026-09-26 14:43:26 on dist-test-slave-x4sn"
server_key: "8fb13c4a28ee0c1649b84019d26c1d96"
server_key_iv: "27e057e4b94218d6f6d8c089ed1e2385"
server_key_version: "encryptionkey@0"
I20260926 14:43:26.075080 3614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:43:26.077566 3629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:26.079020 3614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:43:26.079135 3614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "5015ef0767ae42858a246b45f6e20395"
format_stamp: "Formatted at 2026-09-26 14:43:26 on dist-test-slave-x4sn"
server_key: "8fb13c4a28ee0c1649b84019d26c1d96"
server_key_iv: "27e057e4b94218d6f6d8c089ed1e2385"
server_key_version: "encryptionkey@0"
I20260926 14:43:26.079252 3614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:26.102144 3614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:26.105139 3614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:26.105294 3614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:26.105805 3614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:26.106556 3614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:26.106600 3614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:26.106637 3614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:26.106655 3614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:26.114655 3614 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:41789
I20260926 14:43:26.114679 3743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:41789 every 8 connection(s)
I20260926 14:43:26.115487 3614 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:43:26.118005 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 3614
I20260926 14:43:26.118135 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:43:26.118449 16747 external_mini_cluster.cc:1428] Setting key a59b166002c4263c63926a33f84637bc
WARNING: no policy specified for kudu/127.16.90.194@KRBTEST.COM; defaulting to no policy
Sep 26 14:43:26 dist-test-slave-x4sn krb5kdc[3507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.193@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.194@KRBTEST.COM" created.
I20260926 14:43:26.131989 3591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.117214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:48395 (local address 127.16.90.254:36577)
0926 14:43:26.117385 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:26.117390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:26.120866 (+ 3476us) server_negotiation.cc:407] Connection header received
0926 14:43:26.122468 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:26.122472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:26.122548 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:26.122642 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:26.124188 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.124871 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.125951 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.126201 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.126289 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.128641 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:26.128644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:26.128646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:26.128670 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:26.130032 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:26.130495 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:26.130498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:26.130500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:26.130600 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:26.130934 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:26.130938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:26.130939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:26.131117 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:26.131211 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:26.131693 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 14:43:26.131825 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 14:43:26.132727 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.117626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:36577 (local address 127.16.90.193:48395)
0926 14:43:26.120698 (+ 3072us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:26.120756 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:26.122227 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:26.122939 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:26.122948 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:26.123452 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:26.124032 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.124054 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.124966 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.124968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.125839 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.125847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.126336 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.126340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.126494 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.127051 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:26.127066 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:26.128467 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:26.130141 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:26.130146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:26.130158 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:26.130393 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:26.130694 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:26.130696 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:26.130698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:26.130833 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:26.131212 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:26.131219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:26.131448 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 14:43:26.131600 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 14:43:26.131939 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2841,"thread_start_us":137,"threads_started":1}
I20260926 14:43:26.133980 3744 heartbeater.cc:347] Connected to a master server at 127.16.90.254:36577
I20260926 14:43:26.134199 3744 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:26.134650 3744 heartbeater.cc:511] Master 127.16.90.254:36577 requested a full tablet report, sending...
I20260926 14:43:26.136089 3552 ts_manager.cc:194] Registered new tserver with Master: 5015ef0767ae42858a246b45f6e20395 (127.16.90.193:41789)
I20260926 14:43:26.137174 3552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.193@KRBTEST.COM'} at 127.16.90.193:48395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
Entry for principal kudu/127.16.90.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab.
I20260926 14:43:26.144563 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:36577
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:26.251250 3749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:26.251504 3749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:26.251546 3749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:26.255489 3749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:26.255558 3749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:26.255622 3749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:43:26.260927 3749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.194.keytab
--principal=kudu/127.16.90.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:26.261902 3749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:26.262830 3749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:26.271641 3757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:26.271629 3755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:26.271732 3749 server_base.cc:1034] running on GCE node
W20260926 14:43:26.271629 3754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:26.272305 3749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:26.272893 3749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:26.274046 3749 hybrid_clock.cc:648] HybridClock initialized: now 1790433806274027 us; error 31 us; skew 500 ppm
Sep 26 14:43:26 dist-test-slave-x4sn krb5kdc[3507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:26.276551 3749 init.cc:377] Logged in from keytab as kudu/127.16.90.194@KRBTEST.COM (short username kudu)
I20260926 14:43:26.277375 3749 webserver.cc:460] Webserver started at http://127.16.90.194:43151/ using document root <none> and password file <none>
I20260926 14:43:26.277891 3749 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:26.277954 3749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:26.278144 3749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:26.280128 3749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "a667c043b2aa41199f042aa2d26723f0"
format_stamp: "Formatted at 2026-09-26 14:43:26 on dist-test-slave-x4sn"
server_key: "3c9b927b20dd83c439809da5ae1848dc"
server_key_iv: "6b0c637c82c08721a542491f29f97071"
server_key_version: "encryptionkey@0"
I20260926 14:43:26.280637 3749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "a667c043b2aa41199f042aa2d26723f0"
format_stamp: "Formatted at 2026-09-26 14:43:26 on dist-test-slave-x4sn"
server_key: "3c9b927b20dd83c439809da5ae1848dc"
server_key_iv: "6b0c637c82c08721a542491f29f97071"
server_key_version: "encryptionkey@0"
I20260926 14:43:26.284164 3749 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:43:26.286401 3764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:26.287642 3749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:26.287765 3749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "a667c043b2aa41199f042aa2d26723f0"
format_stamp: "Formatted at 2026-09-26 14:43:26 on dist-test-slave-x4sn"
server_key: "3c9b927b20dd83c439809da5ae1848dc"
server_key_iv: "6b0c637c82c08721a542491f29f97071"
server_key_version: "encryptionkey@0"
I20260926 14:43:26.287873 3749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:26.302378 3749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:26.304951 3749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:26.305111 3749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:26.305660 3749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:26.306423 3749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:26.306468 3749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:26.306512 3749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:26.306541 3749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:26.314841 3749 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:42693
I20260926 14:43:26.314863 3877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:42693 every 8 connection(s)
I20260926 14:43:26.315701 3749 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:43:26.320084 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 3749
I20260926 14:43:26.320222 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:43:26.320520 16747 external_mini_cluster.cc:1428] Setting key 16b1b8510af7a9ee13aab78f843262f6
Sep 26 14:43:26 dist-test-slave-x4sn krb5kdc[3507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.194@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:26.329201 3591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.317364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:39729 (local address 127.16.90.254:36577)
0926 14:43:26.317501 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:26.317505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:26.318275 (+ 770us) server_negotiation.cc:407] Connection header received
0926 14:43:26.319494 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:26.319497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:26.319541 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:26.319613 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:26.321043 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.321578 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.323062 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.323283 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.323357 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.325844 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:26.325847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:26.325849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:26.325872 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:26.327126 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:26.327683 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:26.327686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:26.327688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:26.327765 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:26.328113 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:26.328115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:26.328116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:26.328272 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:26.328340 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:26.328959 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 14:43:26.329082 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.16.90.195@KRBTEST.COM; defaulting to no policy
I20260926 14:43:26.330237 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.317710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:36577 (local address 127.16.90.194:39729)
0926 14:43:26.318118 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:26.318170 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:26.319315 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:26.319796 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:26.319803 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:26.320264 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:26.320898 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.320918 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.321709 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.321712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.322947 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.322958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.323412 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.323416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.323612 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.324174 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:26.324190 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:26.325690 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:26.327234 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:26.327240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:26.327255 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:26.327579 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:26.327842 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:26.327845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:26.327847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:26.328017 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:26.328365 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:26.328372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:26.328675 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 14:43:26.328909 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 14:43:26.329244 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":111,"threads_started":1}
I20260926 14:43:26.331236 3878 heartbeater.cc:347] Connected to a master server at 127.16.90.254:36577
I20260926 14:43:26.331450 3878 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:26.331964 3878 heartbeater.cc:511] Master 127.16.90.254:36577 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.195@KRBTEST.COM" created.
I20260926 14:43:26.332989 3552 ts_manager.cc:194] Registered new tserver with Master: a667c043b2aa41199f042aa2d26723f0 (127.16.90.194:42693)
I20260926 14:43:26.333520 3552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.194@KRBTEST.COM'} at 127.16.90.194:39729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
Entry for principal kudu/127.16.90.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab.
I20260926 14:43:26.344055 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:36577
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:26.450002 3883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:26.450254 3883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:26.450294 3883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:26.454103 3883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:26.454152 3883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:26.454210 3883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:43:26.459297 3883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.195.keytab
--principal=kudu/127.16.90.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:26.460347 3883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:26.461248 3883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:26.470024 3891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:26.470016 3888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:26.470110 3883 server_base.cc:1034] running on GCE node
W20260926 14:43:26.470016 3889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:26.470633 3883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:26.471192 3883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:26.472337 3883 hybrid_clock.cc:648] HybridClock initialized: now 1790433806472320 us; error 28 us; skew 500 ppm
Sep 26 14:43:26 dist-test-slave-x4sn krb5kdc[3507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:26.474817 3883 init.cc:377] Logged in from keytab as kudu/127.16.90.195@KRBTEST.COM (short username kudu)
I20260926 14:43:26.475682 3883 webserver.cc:460] Webserver started at http://127.16.90.195:33251/ using document root <none> and password file <none>
I20260926 14:43:26.476310 3883 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:26.476397 3883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:26.476620 3883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:26.478895 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "829267f71ceb44409c0ac0198d69267e"
format_stamp: "Formatted at 2026-09-26 14:43:26 on dist-test-slave-x4sn"
server_key: "4aae49baa548844715f8e7ac6174694e"
server_key_iv: "5ba420a6d8c73b79b0433f6ff7e6dba8"
server_key_version: "encryptionkey@0"
I20260926 14:43:26.479409 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "829267f71ceb44409c0ac0198d69267e"
format_stamp: "Formatted at 2026-09-26 14:43:26 on dist-test-slave-x4sn"
server_key: "4aae49baa548844715f8e7ac6174694e"
server_key_iv: "5ba420a6d8c73b79b0433f6ff7e6dba8"
server_key_version: "encryptionkey@0"
I20260926 14:43:26.482918 3883 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:43:26.485280 3898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:26.486435 3883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:43:26.486531 3883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "829267f71ceb44409c0ac0198d69267e"
format_stamp: "Formatted at 2026-09-26 14:43:26 on dist-test-slave-x4sn"
server_key: "4aae49baa548844715f8e7ac6174694e"
server_key_iv: "5ba420a6d8c73b79b0433f6ff7e6dba8"
server_key_version: "encryptionkey@0"
I20260926 14:43:26.486624 3883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:26.509883 3883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:26.513083 3883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:26.513264 3883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:26.513845 3883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:26.514643 3883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:26.514689 3883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:26.514729 3883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:26.514757 3883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:26.522970 3883 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:40445
I20260926 14:43:26.523002 4012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:40445 every 8 connection(s)
I20260926 14:43:26.523826 3883 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:43:26.529731 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 3883
I20260926 14:43:26.529857 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:43:26.530151 16747 external_mini_cluster.cc:1428] Setting key 608463908f62ae6d3fd2cd864b5e4364
Sep 26 14:43:26 dist-test-slave-x4sn krb5kdc[3507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433806, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.195@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:26.536772 3591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.525653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:39345 (local address 127.16.90.254:36577)
0926 14:43:26.525789 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:26.525792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:26.526600 (+ 808us) server_negotiation.cc:407] Connection header received
0926 14:43:26.527864 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:26.527867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:26.527916 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:26.528006 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:26.529077 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.529617 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.531064 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.531318 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.531433 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.533787 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:26.533790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:26.533792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:26.533817 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:26.534883 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:26.535343 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:26.535346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:26.535347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:26.535419 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:26.535738 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:26.535742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:26.535744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:26.535917 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:26.536002 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:26.536487 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 14:43:26.536617 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:43:26.537518 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.526036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:36577 (local address 127.16.90.195:39345)
0926 14:43:26.526455 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:26.526497 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:26.527647 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:26.528136 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:26.528143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:26.528470 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:26.528921 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.528933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.529732 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.529735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.530925 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.530935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.531480 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.531485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.531657 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.532249 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:26.532270 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:26.533629 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:26.534983 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:26.534989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:26.535004 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:26.535252 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:26.535497 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:26.535500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:26.535503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:26.535635 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:26.536010 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:26.536016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:26.536260 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 14:43:26.536413 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 14:43:26.536724 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":125,"threads_started":1}
I20260926 14:43:26.538476 4013 heartbeater.cc:347] Connected to a master server at 127.16.90.254:36577
I20260926 14:43:26.538691 4013 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:26.539149 4013 heartbeater.cc:511] Master 127.16.90.254:36577 requested a full tablet report, sending...
I20260926 14:43:26.540103 3552 ts_manager.cc:194] Registered new tserver with Master: 829267f71ceb44409c0ac0198d69267e (127.16.90.195:40445)
I20260926 14:43:26.540649 3552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.90.195@KRBTEST.COM'} at 127.16.90.195:39345
I20260926 14:43:26.543730 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:43:26.551426 3591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.545491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.16.90.254:36577)
0926 14:43:26.545598 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:26.545601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:26.545767 (+ 166us) server_negotiation.cc:407] Connection header received
0926 14:43:26.545860 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:26.545862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:26.545895 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:26.545956 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:26.546526 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.547004 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.547757 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.547958 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.548027 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.549110 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:26.549112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:26.549114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:26.549133 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:26.550318 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:26.550611 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:26.550613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:26.550614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:26.550675 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:26.550836 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:26.550838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:26.550839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:26.551010 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:26.551100 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:26.551239 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 14:43:26.551318 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:43:26.553817 3552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:43:26.556685 3552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:43:26.569911 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.565103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.16.90.195:40445)
0926 14:43:26.565508 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:26.565519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:26.565853 (+ 334us) server_negotiation.cc:407] Connection header received
0926 14:43:26.565919 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:26.565924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:26.566066 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:26.566219 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:26.566655 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.567353 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.568744 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.569294 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.569390 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.569531 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:26.569684 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 14:43:26.569777 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":100,"threads_started":1}
I20260926 14:43:26.569907 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.564388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:40445 (local address 127.0.0.1:34902)
0926 14:43:26.565583 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:26.565595 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:26.565677 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:26.566228 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:26.566234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:26.566256 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:26.566563 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.566570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.567471 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.567474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.568630 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.568639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.569402 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.569405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.569508 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.569543 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 14:43:26.569646 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 14:43:26.569765 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"mutex_wait_us":38,"spinlock_wait_cycles":896,"thread_start_us":82,"threads_started":1}
I20260926 14:43:26.570232 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.564335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:42693 (local address 127.0.0.1:56260)
0926 14:43:26.564963 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:26.565012 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:26.565127 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:26.566245 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:26.566247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:26.566268 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:26.566467 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.566477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.567920 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.567924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.568821 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.568830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.569865 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.569869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.569953 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.569976 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:43:26.570021 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 14:43:26.570088 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":42,"spinlock_wait_cycles":1280,"thread_start_us":62,"threads_started":1}
I20260926 14:43:26.570783 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.564466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.16.90.194:42693)
0926 14:43:26.565883 (+ 1417us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:26.565891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:26.565908 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:43:26.565965 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:26.565969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:26.566104 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:26.566428 (+ 324us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:26.566570 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.567780 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.568928 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.569741 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.570475 (+ 734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.570579 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:26.570615 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 14:43:26.570683 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1336,"thread_start_us":107,"threads_started":1}
I20260926 14:43:26.571699 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.564650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:41789 (local address 127.0.0.1:50792)
0926 14:43:26.565643 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:26.565656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:26.565751 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:26.566357 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:26.566359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:26.566370 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:26.566500 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.566506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.567920 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.567924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.569936 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.569944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.571325 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.571327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.571399 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.571415 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 14:43:26.571471 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:43:26.571536 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":41,"thread_start_us":90,"threads_started":1}
I20260926 14:43:26.572083 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.565097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.16.90.193:41789)
0926 14:43:26.565457 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:26.565467 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:26.565750 (+ 283us) server_negotiation.cc:407] Connection header received
0926 14:43:26.565860 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:26.565866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:26.566066 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:26.566220 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:26.566632 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.567780 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.570172 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.571041 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.571153 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.571274 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:26.571860 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 14:43:26.571956 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":113,"threads_started":1}
I20260926 14:43:26.572916 3947 tablet_service.cc:1467] Processing CreateTablet for tablet 403eb0a3d9a7437888dad309e8344a35 (DEFAULT_TABLE table=test-table [id=b9f4378f33974e61a57849a6d2462705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:26.573163 3812 tablet_service.cc:1467] Processing CreateTablet for tablet 403eb0a3d9a7437888dad309e8344a35 (DEFAULT_TABLE table=test-table [id=b9f4378f33974e61a57849a6d2462705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:26.573778 3947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403eb0a3d9a7437888dad309e8344a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:26.573918 3812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403eb0a3d9a7437888dad309e8344a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:26.578044 3678 tablet_service.cc:1467] Processing CreateTablet for tablet 403eb0a3d9a7437888dad309e8344a35 (DEFAULT_TABLE table=test-table [id=b9f4378f33974e61a57849a6d2462705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:26.579003 3678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403eb0a3d9a7437888dad309e8344a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:26.579715 4030 tablet_bootstrap.cc:492] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e: Bootstrap starting.
I20260926 14:43:26.581174 4031 tablet_bootstrap.cc:492] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0: Bootstrap starting.
I20260926 14:43:26.581748 4030 tablet_bootstrap.cc:654] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:26.582799 4030 log.cc:824] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:26.583408 4031 tablet_bootstrap.cc:654] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:26.584522 4030 tablet_bootstrap.cc:492] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e: No bootstrap required, opened a new log
I20260926 14:43:26.584700 4030 ts_tablet_manager.cc:1397] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:43:26.584684 4031 log.cc:824] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:26.585997 4034 tablet_bootstrap.cc:492] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395: Bootstrap starting.
I20260926 14:43:26.586550 4031 tablet_bootstrap.cc:492] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0: No bootstrap required, opened a new log
I20260926 14:43:26.586714 4031 ts_tablet_manager.cc:1397] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:43:26.588239 4034 tablet_bootstrap.cc:654] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:26.588727 4030 raft_consensus.cc:348] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a667c043b2aa41199f042aa2d26723f0" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 42693 } } peers { permanent_uuid: "829267f71ceb44409c0ac0198d69267e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 40445 } } peers { permanent_uuid: "5015ef0767ae42858a246b45f6e20395" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 41789 } }
I20260926 14:43:26.588996 4030 raft_consensus.cc:374] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:26.589043 4030 raft_consensus.cc:729] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829267f71ceb44409c0ac0198d69267e, State: Initialized, Role: FOLLOWER
I20260926 14:43:26.589445 4034 log.cc:824] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:26.589567 4030 consensus_queue.cc:260] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a667c043b2aa41199f042aa2d26723f0" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 42693 } } peers { permanent_uuid: "829267f71ceb44409c0ac0198d69267e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 40445 } } peers { permanent_uuid: "5015ef0767ae42858a246b45f6e20395" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 41789 } }
I20260926 14:43:26.590199 4013 heartbeater.cc:503] Master 127.16.90.254:36577 was elected leader, sending a full tablet report...
I20260926 14:43:26.590018 4031 raft_consensus.cc:348] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a667c043b2aa41199f042aa2d26723f0" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 42693 } } peers { permanent_uuid: "829267f71ceb44409c0ac0198d69267e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 40445 } } peers { permanent_uuid: "5015ef0767ae42858a246b45f6e20395" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 41789 } }
I20260926 14:43:26.590389 4031 raft_consensus.cc:374] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:26.590447 4031 raft_consensus.cc:729] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a667c043b2aa41199f042aa2d26723f0, State: Initialized, Role: FOLLOWER
I20260926 14:43:26.590541 4030 ts_tablet_manager.cc:1428] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 14:43:26.590843 4031 consensus_queue.cc:260] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a667c043b2aa41199f042aa2d26723f0" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 42693 } } peers { permanent_uuid: "829267f71ceb44409c0ac0198d69267e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 40445 } } peers { permanent_uuid: "5015ef0767ae42858a246b45f6e20395" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 41789 } }
I20260926 14:43:26.591811 4031 ts_tablet_manager.cc:1428] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:43:26.592098 3878 heartbeater.cc:503] Master 127.16.90.254:36577 was elected leader, sending a full tablet report...
I20260926 14:43:26.592316 4034 tablet_bootstrap.cc:492] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395: No bootstrap required, opened a new log
I20260926 14:43:26.592491 4034 ts_tablet_manager.cc:1397] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 14:43:26.596086 4034 raft_consensus.cc:348] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a667c043b2aa41199f042aa2d26723f0" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 42693 } } peers { permanent_uuid: "829267f71ceb44409c0ac0198d69267e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 40445 } } peers { permanent_uuid: "5015ef0767ae42858a246b45f6e20395" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 41789 } }
I20260926 14:43:26.596279 4034 raft_consensus.cc:374] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:26.596314 4034 raft_consensus.cc:729] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5015ef0767ae42858a246b45f6e20395, State: Initialized, Role: FOLLOWER
I20260926 14:43:26.596601 4034 consensus_queue.cc:260] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a667c043b2aa41199f042aa2d26723f0" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 42693 } } peers { permanent_uuid: "829267f71ceb44409c0ac0198d69267e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 40445 } } peers { permanent_uuid: "5015ef0767ae42858a246b45f6e20395" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 41789 } }
I20260926 14:43:26.597129 3744 heartbeater.cc:503] Master 127.16.90.254:36577 was elected leader, sending a full tablet report...
I20260926 14:43:26.597388 4034 ts_tablet_manager.cc:1428] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 14:43:26.616570 3745 tablet.cc:2369] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:43:26.634234 4037 raft_consensus.cc:482] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:43:26.634429 4037 raft_consensus.cc:504] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a667c043b2aa41199f042aa2d26723f0" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 42693 } } peers { permanent_uuid: "829267f71ceb44409c0ac0198d69267e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 40445 } } peers { permanent_uuid: "5015ef0767ae42858a246b45f6e20395" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 41789 } }
I20260926 14:43:26.635545 4037 leader_election.cc:288] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 829267f71ceb44409c0ac0198d69267e (127.16.90.195:40445), 5015ef0767ae42858a246b45f6e20395 (127.16.90.193:41789)
I20260926 14:43:26.638970 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.635972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:51745 (local address 127.16.90.193:41789)
0926 14:43:26.636087 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:26.636092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:26.636130 (+ 38us) server_negotiation.cc:407] Connection header received
0926 14:43:26.636295 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:26.636298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:26.636341 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:26.636401 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:26.636647 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.637129 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.638103 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.638478 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.638578 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.638630 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:26.638799 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 14:43:26.638866 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:43:26.638970 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.635818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:41789 (local address 127.16.90.194:51745)
0926 14:43:26.636055 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:26.636070 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:26.636186 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:26.636418 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:26.636421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:26.636437 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:26.636545 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.636550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.637226 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.637230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.637983 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.637991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.638578 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.638582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.638688 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.638707 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:43:26.638769 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 14:43:26.638839 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260926 14:43:26.639180 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.635915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:52219 (local address 127.16.90.195:40445)
0926 14:43:26.636063 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:26.636068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:26.636317 (+ 249us) server_negotiation.cc:407] Connection header received
0926 14:43:26.636436 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:26.636438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:26.636482 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:26.636551 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:26.636810 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.637296 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.638295 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.638691 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.638793 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.638841 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:26.639003 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 14:43:26.639069 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:43:26.639210 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.635818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:40445 (local address 127.16.90.194:52219)
0926 14:43:26.636214 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:26.636226 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:26.636337 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:26.636567 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:26.636570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:26.636585 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:26.636712 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.636718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.637399 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.637402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.638175 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:26.638183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.638794 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.638797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:26.638893 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.638914 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:43:26.638980 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:43:26.639055 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":53,"thread_start_us":139,"threads_started":1}
I20260926 14:43:26.639717 3698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403eb0a3d9a7437888dad309e8344a35" candidate_uuid: "a667c043b2aa41199f042aa2d26723f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5015ef0767ae42858a246b45f6e20395" is_pre_election: true
I20260926 14:43:26.639750 3967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403eb0a3d9a7437888dad309e8344a35" candidate_uuid: "a667c043b2aa41199f042aa2d26723f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829267f71ceb44409c0ac0198d69267e" is_pre_election: true
I20260926 14:43:26.639992 3967 raft_consensus.cc:2454] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a667c043b2aa41199f042aa2d26723f0 in term 0.
I20260926 14:43:26.639995 3698 raft_consensus.cc:2454] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a667c043b2aa41199f042aa2d26723f0 in term 0.
I20260926 14:43:26.640430 3767 leader_election.cc:302] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 829267f71ceb44409c0ac0198d69267e, a667c043b2aa41199f042aa2d26723f0; no voters:
I20260926 14:43:26.640684 4037 raft_consensus.cc:2781] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:43:26.640758 4037 raft_consensus.cc:482] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:43:26.640808 4037 raft_consensus.cc:3037] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:26.641799 4037 raft_consensus.cc:504] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a667c043b2aa41199f042aa2d26723f0" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 42693 } } peers { permanent_uuid: "829267f71ceb44409c0ac0198d69267e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 40445 } } peers { permanent_uuid: "5015ef0767ae42858a246b45f6e20395" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 41789 } }
I20260926 14:43:26.642127 4037 leader_election.cc:288] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [CANDIDATE]: Term 1 election: Requested vote from peers 829267f71ceb44409c0ac0198d69267e (127.16.90.195:40445), 5015ef0767ae42858a246b45f6e20395 (127.16.90.193:41789)
I20260926 14:43:26.642428 3967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403eb0a3d9a7437888dad309e8344a35" candidate_uuid: "a667c043b2aa41199f042aa2d26723f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829267f71ceb44409c0ac0198d69267e"
I20260926 14:43:26.642546 3967 raft_consensus.cc:3037] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:26.643530 3967 raft_consensus.cc:2454] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a667c043b2aa41199f042aa2d26723f0 in term 1.
I20260926 14:43:26.643803 3767 leader_election.cc:302] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 829267f71ceb44409c0ac0198d69267e, a667c043b2aa41199f042aa2d26723f0; no voters:
I20260926 14:43:26.643817 3698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403eb0a3d9a7437888dad309e8344a35" candidate_uuid: "a667c043b2aa41199f042aa2d26723f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5015ef0767ae42858a246b45f6e20395"
I20260926 14:43:26.643923 3698 raft_consensus.cc:3037] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:26.643966 4037 raft_consensus.cc:2781] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:26.644158 4037 raft_consensus.cc:686] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 1 LEADER]: Becoming Leader. State: Replica: a667c043b2aa41199f042aa2d26723f0, State: Running, Role: LEADER
I20260926 14:43:26.644471 4037 consensus_queue.cc:237] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a667c043b2aa41199f042aa2d26723f0" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 42693 } } peers { permanent_uuid: "829267f71ceb44409c0ac0198d69267e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 40445 } } peers { permanent_uuid: "5015ef0767ae42858a246b45f6e20395" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 41789 } }
I20260926 14:43:26.644992 3698 raft_consensus.cc:2454] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a667c043b2aa41199f042aa2d26723f0 in term 1.
I20260926 14:43:26.647636 3551 catalog_manager.cc:5495] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a667c043b2aa41199f042aa2d26723f0 (127.16.90.194). New cstate: current_term: 1 leader_uuid: "a667c043b2aa41199f042aa2d26723f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a667c043b2aa41199f042aa2d26723f0" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "829267f71ceb44409c0ac0198d69267e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 40445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5015ef0767ae42858a246b45f6e20395" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 41789 } health_report { overall_health: UNKNOWN } } }
I20260926 14:43:26.672163 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.669093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.16.90.194:42693)
0926 14:43:26.669213 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:26.669217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:26.669230 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:43:26.669343 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:26.669345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:26.669401 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:26.669487 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:26.669680 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.670176 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.670920 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.671134 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.671201 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.671422 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:26.671850 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:26.671937 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:26.671965 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 14:43:26.672013 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:43:26.677659 3698 raft_consensus.cc:1261] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395 [term 1 FOLLOWER]: Refusing update from remote peer a667c043b2aa41199f042aa2d26723f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:26.677665 3967 raft_consensus.cc:1261] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e [term 1 FOLLOWER]: Refusing update from remote peer a667c043b2aa41199f042aa2d26723f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:26.678314 4040 consensus_queue.cc:1035] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829267f71ceb44409c0ac0198d69267e" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 40445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:26.678483 4037 consensus_queue.cc:1035] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5015ef0767ae42858a246b45f6e20395" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 41789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:26.685827 4045 mvcc.cc:205] Tried to move back new op lower bound from 7333616872145055744 to 7333616872018341888. Current Snapshot: MvccSnapshot[applied={T|T < 7333616872145055744}]
I20260926 14:43:26.685978 4044 mvcc.cc:205] Tried to move back new op lower bound from 7333616872145055744 to 7333616872018341888. Current Snapshot: MvccSnapshot[applied={T|T < 7333616872145055744}]
I20260926 14:43:26.695312 3551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:43:26.695484 3551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:43:26.697804 3551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1431d46499fd45cc988e338bba1e8072: Sending DeleteTablet for 3 replicas of tablet 403eb0a3d9a7437888dad309e8344a35
I20260926 14:43:26.698479 3678 tablet_service.cc:1514] Processing DeleteTablet for tablet 403eb0a3d9a7437888dad309e8344a35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:26 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:50792
I20260926 14:43:26.698462 3812 tablet_service.cc:1514] Processing DeleteTablet for tablet 403eb0a3d9a7437888dad309e8344a35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:26 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:56260
I20260926 14:43:26.698944 4054 tablet_replica.cc:330] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395: stopping tablet replica
I20260926 14:43:26.698930 3947 tablet_service.cc:1514] Processing DeleteTablet for tablet 403eb0a3d9a7437888dad309e8344a35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:26 UTC) from {username='kudu', principal='kudu/127.16.90.254@KRBTEST.COM'} at 127.0.0.1:34902
I20260926 14:43:26.699255 4054 raft_consensus.cc:2229] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:43:26.699389 4055 tablet_replica.cc:330] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e: stopping tablet replica
I20260926 14:43:26.699558 4054 raft_consensus.cc:2258] T 403eb0a3d9a7437888dad309e8344a35 P 5015ef0767ae42858a246b45f6e20395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:43:26.699884 4055 raft_consensus.cc:2229] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:43:26.700194 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 3614
I20260926 14:43:26.700786 4053 tablet_replica.cc:330] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0: stopping tablet replica
I20260926 14:43:26.701051 4053 raft_consensus.cc:2229] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:43:26.701414 4053 raft_consensus.cc:2258] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:43:26.702827 4053 ts_tablet_manager.cc:1905] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:43:26.705611 4053 ts_tablet_manager.cc:1918] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:43:26.705708 4053 log.cc:1196] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/403eb0a3d9a7437888dad309e8344a35
I20260926 14:43:26.705993 4053 ts_tablet_manager.cc:1939] T 403eb0a3d9a7437888dad309e8344a35 P a667c043b2aa41199f042aa2d26723f0: Deleting consensus metadata
I20260926 14:43:26.706136 4055 raft_consensus.cc:2258] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:43:26.706573 3536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:41789 (error 108)
I20260926 14:43:26.707284 3537 catalog_manager.cc:4841] TS a667c043b2aa41199f042aa2d26723f0 (127.16.90.194:42693): tablet 403eb0a3d9a7437888dad309e8344a35 (table test-table [id=b9f4378f33974e61a57849a6d2462705]) successfully deleted
I20260926 14:43:26.707451 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 3749
I20260926 14:43:26.707602 4055 ts_tablet_manager.cc:1905] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:43:26.710470 4055 ts_tablet_manager.cc:1918] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:43:26.710559 4055 log.cc:1196] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/403eb0a3d9a7437888dad309e8344a35
I20260926 14:43:26.710896 4055 ts_tablet_manager.cc:1939] T 403eb0a3d9a7437888dad309e8344a35 P 829267f71ceb44409c0ac0198d69267e: Deleting consensus metadata
I20260926 14:43:26.711869 3538 catalog_manager.cc:4841] TS 829267f71ceb44409c0ac0198d69267e (127.16.90.195:40445): tablet 403eb0a3d9a7437888dad309e8344a35 (table test-table [id=b9f4378f33974e61a57849a6d2462705]) successfully deleted
I20260926 14:43:26.712482 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.712187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:41789 (local address 127.0.0.1:50804)
0926 14:43:26.712320 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:26.712398 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:41789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:43:26.712630 3536 catalog_manager.cc:4584] TS 5015ef0767ae42858a246b45f6e20395 (127.16.90.193:41789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 403eb0a3d9a7437888dad309e8344a35: Network error: Client connection negotiation failed: client connection to 127.16.90.193:41789: connect: Connection refused (error 111)
I20260926 14:43:26.714525 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 3883
I20260926 14:43:26.718997 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 3520
2026-09-26T14:43:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2141 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T14:43:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:26Z Disabled control of system clock
I20260926 14:43:26.746444 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35747
--webserver_interface=127.16.90.254
--webserver_port=0
--builtin_ntp_servers=127.16.90.212:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:35747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:43:26.851802 4060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:26.852061 4060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:26.852100 4060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:26.856067 4060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:26.856120 4060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:26.856137 4060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:26.856165 4060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:26.856191 4060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:26.860812 4060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:35747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:26.861846 4060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:26.863178 4060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:26.870894 4066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:26.870941 4065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:26.870947 4068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:26.871057 4060 server_base.cc:1034] running on GCE node
I20260926 14:43:26.871762 4060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:26.872800 4060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:26.873955 4060 hybrid_clock.cc:648] HybridClock initialized: now 1790433806873941 us; error 29 us; skew 500 ppm
I20260926 14:43:26.875707 4060 webserver.cc:460] Webserver started at http://127.16.90.254:40999/ using document root <none> and password file <none>
I20260926 14:43:26.876246 4060 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:26.876312 4060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:26.876511 4060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:26.878435 4060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "49e95646655f403e81b5599e051795b9"
format_stamp: "Formatted at 2026-09-26 14:43:26 on dist-test-slave-x4sn"
server_key: "e5c423b1cab69848e026e392b8d166b5"
server_key_iv: "b3422f4f0a7619f348f84fcb81d679e1"
server_key_version: "encryptionkey@0"
I20260926 14:43:26.878945 4060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "49e95646655f403e81b5599e051795b9"
format_stamp: "Formatted at 2026-09-26 14:43:26 on dist-test-slave-x4sn"
server_key: "e5c423b1cab69848e026e392b8d166b5"
server_key_iv: "b3422f4f0a7619f348f84fcb81d679e1"
server_key_version: "encryptionkey@0"
I20260926 14:43:26.882453 4060 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 14:43:26.884732 4074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:26.885833 4060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:43:26.885939 4060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "49e95646655f403e81b5599e051795b9"
format_stamp: "Formatted at 2026-09-26 14:43:26 on dist-test-slave-x4sn"
server_key: "e5c423b1cab69848e026e392b8d166b5"
server_key_iv: "b3422f4f0a7619f348f84fcb81d679e1"
server_key_version: "encryptionkey@0"
I20260926 14:43:26.886029 4060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:26.907861 4060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:26.908584 4060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:26.908761 4060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:26.915512 4060 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:35747
I20260926 14:43:26.915527 4126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:35747 every 8 connection(s)
I20260926 14:43:26.916435 4060 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:26.919373 4127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:26.921519 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 4060
I20260926 14:43:26.921622 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:26.921890 16747 external_mini_cluster.cc:1428] Setting key cfee099be09cb262ca0cc9b892fb4c9f
I20260926 14:43:26.925355 4127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9: Bootstrap starting.
I20260926 14:43:26.927964 4127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:26.928763 4127 log.cc:824] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:26.930861 4127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9: No bootstrap required, opened a new log
I20260926 14:43:26.932010 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:26.923229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40992 (local address 127.16.90.254:35747)
0926 14:43:26.923923 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:26.923942 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:26.923986 (+ 44us) server_negotiation.cc:407] Connection header received
0926 14:43:26.924785 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:26.924808 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:26.925291 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:26.925654 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:26.926089 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.927545 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.928441 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:26.929122 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:26.929268 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:26.930037 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:26.930043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:26.930058 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:26.930524 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:26.930612 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:26.930639 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:26.930736 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:26.930823 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 14:43:26.931080 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":106,"threads_started":1}
I20260926 14:43:26.934763 4127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e95646655f403e81b5599e051795b9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35747 } }
I20260926 14:43:26.934968 4127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:26.935012 4127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e95646655f403e81b5599e051795b9, State: Initialized, Role: FOLLOWER
I20260926 14:43:26.935425 4127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e95646655f403e81b5599e051795b9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35747 } }
I20260926 14:43:26.935606 4127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:26.935719 4127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:26.935824 4127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:26.936717 4127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e95646655f403e81b5599e051795b9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35747 } }
I20260926 14:43:26.937038 4127 leader_election.cc:302] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49e95646655f403e81b5599e051795b9; no voters:
I20260926 14:43:26.937300 4127 leader_election.cc:288] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:26.937381 4132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:26.937563 4132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [term 1 LEADER]: Becoming Leader. State: Replica: 49e95646655f403e81b5599e051795b9, State: Running, Role: LEADER
I20260926 14:43:26.937844 4132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e95646655f403e81b5599e051795b9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35747 } }
I20260926 14:43:26.938458 4127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:26.939334 4134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49e95646655f403e81b5599e051795b9. Latest consensus state: current_term: 1 leader_uuid: "49e95646655f403e81b5599e051795b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e95646655f403e81b5599e051795b9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35747 } } }
I20260926 14:43:26.939458 4134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:26.939842 4141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:26.939831 4133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49e95646655f403e81b5599e051795b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e95646655f403e81b5599e051795b9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35747 } } }
I20260926 14:43:26.940032 4133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:26.942440 4141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:26.947631 4141 catalog_manager.cc:1347] Generated new cluster ID: 26cd1f67b53c4b29b3b1e1b860c52b9f
I20260926 14:43:26.947729 4141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:26.974776 4141 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:26.976094 4141 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:26.987656 4141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9: Generated new TSK 0
I20260926 14:43:26.988485 4141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:26.997237 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.193:0
--local_ip_for_outbound_sockets=127.16.90.193
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35747
--builtin_ntp_servers=127.16.90.212:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:43:27.104818 4152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:27.105059 4152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:27.105094 4152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:27.108994 4152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:27.109042 4152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:27.109093 4152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.193
I20260926 14:43:27.113638 4152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.90.193
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.193
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:27.114590 4152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:27.115899 4152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:27.126405 4158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:27.126422 4157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:27.126770 4160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:27.127117 4152 server_base.cc:1034] running on GCE node
I20260926 14:43:27.127480 4152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:27.128185 4152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:27.129356 4152 hybrid_clock.cc:648] HybridClock initialized: now 1790433807129327 us; error 44 us; skew 500 ppm
I20260926 14:43:27.131439 4152 webserver.cc:460] Webserver started at http://127.16.90.193:44885/ using document root <none> and password file <none>
I20260926 14:43:27.132154 4152 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:27.132241 4152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:27.132477 4152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:27.135320 4152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/data/instance:
uuid: "11213448e6104ad68db8ae9ad0353822"
format_stamp: "Formatted at 2026-09-26 14:43:27 on dist-test-slave-x4sn"
server_key: "8aca541c941811017094928ad4dd1c00"
server_key_iv: "26e1a11b57552bc506312979b3210e9d"
server_key_version: "encryptionkey@0"
I20260926 14:43:27.136016 4152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance:
uuid: "11213448e6104ad68db8ae9ad0353822"
format_stamp: "Formatted at 2026-09-26 14:43:27 on dist-test-slave-x4sn"
server_key: "8aca541c941811017094928ad4dd1c00"
server_key_iv: "26e1a11b57552bc506312979b3210e9d"
server_key_version: "encryptionkey@0"
I20260926 14:43:27.140769 4152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:43:27.143075 4166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:27.144315 4152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:27.144428 4152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/wal
uuid: "11213448e6104ad68db8ae9ad0353822"
format_stamp: "Formatted at 2026-09-26 14:43:27 on dist-test-slave-x4sn"
server_key: "8aca541c941811017094928ad4dd1c00"
server_key_iv: "26e1a11b57552bc506312979b3210e9d"
server_key_version: "encryptionkey@0"
I20260926 14:43:27.144524 4152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:27.157215 4152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:27.157830 4152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:27.157991 4152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:27.158536 4152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:27.159410 4152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:27.159456 4152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:27.159507 4152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:27.159531 4152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:27.167435 4152 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.193:39107
I20260926 14:43:27.167469 4279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.193:39107 every 8 connection(s)
I20260926 14:43:27.168303 4152 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/data/info.pb
I20260926 14:43:27.173075 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 4152
I20260926 14:43:27.173182 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-0/wal/instance
I20260926 14:43:27.173434 16747 external_mini_cluster.cc:1428] Setting key a0e07e36be323b2b5abeb8a0fef7362a
I20260926 14:43:27.175174 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.194:0
--local_ip_for_outbound_sockets=127.16.90.194
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35747
--builtin_ntp_servers=127.16.90.212:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:43:27.178282 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.170009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.193:44929 (local address 127.16.90.254:35747)
0926 14:43:27.170149 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:27.170154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:27.171197 (+ 1043us) server_negotiation.cc:407] Connection header received
0926 14:43:27.172544 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:27.172547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:27.172594 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:27.172668 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:27.173606 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.174131 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.175242 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.175501 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.175599 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.177843 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:27.177847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:27.177850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:27.177863 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:27.177881 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:27.177888 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:27.177946 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:27.178086 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 14:43:27.178146 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:43:27.179234 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.170425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35747 (local address 127.16.90.193:44929)
0926 14:43:27.171039 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:27.171089 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:27.172368 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:27.172815 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:27.172821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:27.172904 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:27.173438 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.173452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.174255 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.174258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.175124 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.175133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.175643 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.175647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.175826 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.177591 (+ 1765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:43:27.177593 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:27.177633 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:43:27.177696 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:27.177955 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:27.177963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:27.177970 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 14:43:27.178086 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 14:43:27.178486 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
I20260926 14:43:27.180480 4280 heartbeater.cc:347] Connected to a master server at 127.16.90.254:35747
I20260926 14:43:27.180693 4280 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:27.181166 4280 heartbeater.cc:511] Master 127.16.90.254:35747 requested a full tablet report, sending...
I20260926 14:43:27.182564 4091 ts_manager.cc:194] Registered new tserver with Master: 11213448e6104ad68db8ae9ad0353822 (127.16.90.193:39107)
I20260926 14:43:27.183630 4091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.90.193:44929
W20260926 14:43:27.280715 4284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:27.280947 4284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:27.280999 4284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:27.284891 4284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:27.284937 4284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:27.285007 4284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.194
I20260926 14:43:27.289691 4284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.90.194
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.194
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:27.290683 4284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:27.292047 4284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:27.300866 4290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:27.300910 4292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:27.300884 4289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:27.301076 4284 server_base.cc:1034] running on GCE node
I20260926 14:43:27.301555 4284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:27.302146 4284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:27.303304 4284 hybrid_clock.cc:648] HybridClock initialized: now 1790433807303289 us; error 31 us; skew 500 ppm
I20260926 14:43:27.304977 4284 webserver.cc:460] Webserver started at http://127.16.90.194:44361/ using document root <none> and password file <none>
I20260926 14:43:27.305506 4284 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:27.305569 4284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:27.305763 4284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:27.307761 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/data/instance:
uuid: "41175da4854c497ba5629943428f9f1c"
format_stamp: "Formatted at 2026-09-26 14:43:27 on dist-test-slave-x4sn"
server_key: "c2b358fdc30e8606f329b1a359cddfc3"
server_key_iv: "dc21e6ca5f27c8d3b58b3e83e68f021e"
server_key_version: "encryptionkey@0"
I20260926 14:43:27.308290 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance:
uuid: "41175da4854c497ba5629943428f9f1c"
format_stamp: "Formatted at 2026-09-26 14:43:27 on dist-test-slave-x4sn"
server_key: "c2b358fdc30e8606f329b1a359cddfc3"
server_key_iv: "dc21e6ca5f27c8d3b58b3e83e68f021e"
server_key_version: "encryptionkey@0"
I20260926 14:43:27.311764 4284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:27.313962 4298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:27.315083 4284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:27.315191 4284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/wal
uuid: "41175da4854c497ba5629943428f9f1c"
format_stamp: "Formatted at 2026-09-26 14:43:27 on dist-test-slave-x4sn"
server_key: "c2b358fdc30e8606f329b1a359cddfc3"
server_key_iv: "dc21e6ca5f27c8d3b58b3e83e68f021e"
server_key_version: "encryptionkey@0"
I20260926 14:43:27.315279 4284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:27.325392 4284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:27.325953 4284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:27.326100 4284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:27.326606 4284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:27.327440 4284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:27.327485 4284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:27.327533 4284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:27.327561 4284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:27.335373 4284 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.194:46323
I20260926 14:43:27.335403 4413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.194:46323 every 8 connection(s)
I20260926 14:43:27.336201 4284 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/data/info.pb
I20260926 14:43:27.340763 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 4284
I20260926 14:43:27.340875 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/wal/instance
I20260926 14:43:27.341135 16747 external_mini_cluster.cc:1428] Setting key e89972d7e924ac2cd9039b8973e7f5e9
I20260926 14:43:27.342926 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.90.195:0
--local_ip_for_outbound_sockets=127.16.90.195
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35747
--builtin_ntp_servers=127.16.90.212:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:43:27.345081 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.337793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:37717 (local address 127.16.90.254:35747)
0926 14:43:27.337929 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:27.337934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:27.338981 (+ 1047us) server_negotiation.cc:407] Connection header received
0926 14:43:27.340412 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:27.340414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:27.340460 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:27.340530 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:27.341453 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.341985 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.343223 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.343522 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.343620 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.344696 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:27.344699 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:27.344701 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:27.344712 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:27.344730 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:27.344738 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:27.344793 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:27.344931 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 14:43:27.344978 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:43:27.345974 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.338231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35747 (local address 127.16.90.194:37717)
0926 14:43:27.338847 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:27.338884 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:27.340227 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:27.340694 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:27.340701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:27.340783 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:27.341301 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.341315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.342103 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.342106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.343110 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.343122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.343651 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.343654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.343821 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.344417 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:43:27.344419 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:27.344472 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:43:27.344550 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:27.344808 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:27.344814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:27.344820 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:43:27.344934 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 14:43:27.345217 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
I20260926 14:43:27.346963 4414 heartbeater.cc:347] Connected to a master server at 127.16.90.254:35747
I20260926 14:43:27.347173 4414 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:27.347626 4414 heartbeater.cc:511] Master 127.16.90.254:35747 requested a full tablet report, sending...
I20260926 14:43:27.348584 4091 ts_manager.cc:194] Registered new tserver with Master: 41175da4854c497ba5629943428f9f1c (127.16.90.194:46323)
I20260926 14:43:27.349124 4091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.90.194:37717
W20260926 14:43:27.449347 4417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:27.449584 4417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:27.449635 4417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:27.453509 4417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:27.453555 4417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:27.453626 4417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.90.195
I20260926 14:43:27.458257 4417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.90.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.90.195
--webserver_port=0
--tserver_master_addrs=127.16.90.254:35747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.90.195
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:27.459268 4417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:27.460644 4417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:27.469390 4422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:27.469436 4425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:27.469406 4423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:27.469655 4417 server_base.cc:1034] running on GCE node
I20260926 14:43:27.470019 4417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:27.470577 4417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:27.471740 4417 hybrid_clock.cc:648] HybridClock initialized: now 1790433807471711 us; error 39 us; skew 500 ppm
I20260926 14:43:27.473399 4417 webserver.cc:460] Webserver started at http://127.16.90.195:38101/ using document root <none> and password file <none>
I20260926 14:43:27.473920 4417 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:27.473985 4417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:27.474172 4417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:27.476128 4417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/data/instance:
uuid: "499830e7327e4a00888553bc4943de68"
format_stamp: "Formatted at 2026-09-26 14:43:27 on dist-test-slave-x4sn"
server_key: "a11db293e8d9bc23c97809c983595929"
server_key_iv: "dece9d8cdde2d2ceff487e56ff2298fe"
server_key_version: "encryptionkey@0"
I20260926 14:43:27.476631 4417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance:
uuid: "499830e7327e4a00888553bc4943de68"
format_stamp: "Formatted at 2026-09-26 14:43:27 on dist-test-slave-x4sn"
server_key: "a11db293e8d9bc23c97809c983595929"
server_key_iv: "dece9d8cdde2d2ceff487e56ff2298fe"
server_key_version: "encryptionkey@0"
I20260926 14:43:27.480590 4417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:43:27.483062 4431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:27.484261 4417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 14:43:27.484362 4417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/wal
uuid: "499830e7327e4a00888553bc4943de68"
format_stamp: "Formatted at 2026-09-26 14:43:27 on dist-test-slave-x4sn"
server_key: "a11db293e8d9bc23c97809c983595929"
server_key_iv: "dece9d8cdde2d2ceff487e56ff2298fe"
server_key_version: "encryptionkey@0"
I20260926 14:43:27.484449 4417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:27.502557 4417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:27.503219 4417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:27.503376 4417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:27.503964 4417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:43:27.504849 4417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:43:27.504897 4417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:27.504947 4417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:43:27.504972 4417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:27.512962 4417 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.195:43941
I20260926 14:43:27.512991 4545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.195:43941 every 8 connection(s)
I20260926 14:43:27.513793 4417 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/data/info.pb
I20260926 14:43:27.519279 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 4417
I20260926 14:43:27.519394 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/wal/instance
I20260926 14:43:27.519706 16747 external_mini_cluster.cc:1428] Setting key 8b3798b9c2f39609e35223e3a9737303
I20260926 14:43:27.522915 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.515485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.195:42385 (local address 127.16.90.254:35747)
0926 14:43:27.515615 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:27.515619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:27.516610 (+ 991us) server_negotiation.cc:407] Connection header received
0926 14:43:27.517931 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:27.517933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:27.517980 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:27.518050 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:27.518875 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.519606 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.521029 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.521375 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.521475 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.522490 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:27.522493 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:27.522495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:27.522507 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:27.522528 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:27.522535 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:27.522591 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:27.522761 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 14:43:27.522810 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:43:27.523798 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.515852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:35747 (local address 127.16.90.195:42385)
0926 14:43:27.516463 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:27.516502 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:27.517743 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:27.518180 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:27.518186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:27.518268 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:27.518739 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.518751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.519782 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.519786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.520915 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.520926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.521550 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.521555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.521740 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.522250 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:43:27.522252 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:27.522290 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:43:27.522353 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:27.522625 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:27.522634 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:27.522640 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:43:27.522759 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 14:43:27.522995 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260926 14:43:27.524817 4546 heartbeater.cc:347] Connected to a master server at 127.16.90.254:35747
I20260926 14:43:27.525027 4546 heartbeater.cc:464] Registering TS with master...
I20260926 14:43:27.525513 4546 heartbeater.cc:511] Master 127.16.90.254:35747 requested a full tablet report, sending...
I20260926 14:43:27.526443 4091 ts_manager.cc:194] Registered new tserver with Master: 499830e7327e4a00888553bc4943de68 (127.16.90.195:43941)
I20260926 14:43:27.526983 4091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.90.195:42385
I20260926 14:43:27.533295 16747 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:43:27.538267 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.535117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.16.90.254:35747)
0926 14:43:27.535228 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:27.535231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:27.535393 (+ 162us) server_negotiation.cc:407] Connection header received
0926 14:43:27.535487 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:27.535489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:27.535527 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:27.535587 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:27.535842 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.536346 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.537098 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.537297 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.537363 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.537984 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:27.537987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:27.537989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:27.537999 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:27.538018 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:27.538025 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:27.538076 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:27.538116 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 14:43:27.538164 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:43:27.540977 4091 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:43:27.543893 4091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:43:27.557588 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.551871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.16.90.193:39107)
0926 14:43:27.552234 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:27.552246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:27.552875 (+ 629us) server_negotiation.cc:407] Connection header received
0926 14:43:27.553084 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:27.553091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:27.553302 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:27.553417 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:27.553934 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.554792 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.556068 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.556844 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.556900 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.557035 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:27.557349 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 14:43:27.557442 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":158,"threads_started":1}
I20260926 14:43:27.557570 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.551761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:39107 (local address 127.0.0.1:60214)
0926 14:43:27.552375 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:27.552799 (+ 424us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:27.552892 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:27.553581 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:27.553586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:27.553622 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:27.553830 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.553836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.554889 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.554892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.555935 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.555944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.557112 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.557114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.557189 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.557213 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:43:27.557298 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:43:27.557415 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":68,"spinlock_wait_cycles":6144,"thread_start_us":55,"threads_started":1}
I20260926 14:43:27.557919 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.551857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:43941 (local address 127.0.0.1:45332)
0926 14:43:27.552651 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:27.552712 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:27.552833 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:27.554219 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:27.554222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:27.554240 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:27.554312 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.554317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.555525 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.555528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.556626 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.556636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.557569 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.557573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.557656 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.557677 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:43:27.557725 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 14:43:27.557789 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":105,"threads_started":1}
I20260926 14:43:27.558153 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.552431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.194:46323 (local address 127.0.0.1:44082)
0926 14:43:27.553032 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:27.553044 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:27.553111 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:27.553581 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:27.553586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:27.553620 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:27.553841 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.553849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.555223 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.555227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.556541 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.556550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.557792 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.557795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.557873 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.557892 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:43:27.557966 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 14:43:27.558029 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":39,"spinlock_wait_cycles":6272,"thread_start_us":90,"threads_started":1}
I20260926 14:43:27.558275 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.552574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.16.90.194:46323)
0926 14:43:27.553188 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:27.553200 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:27.553223 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:43:27.553294 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:27.553300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:27.553447 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:27.553555 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:27.553966 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.555086 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.556666 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.557433 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.557538 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.557734 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:27.558003 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 14:43:27.558126 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":115,"threads_started":1}
I20260926 14:43:27.558905 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.552495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.16.90.195:43941)
0926 14:43:27.553820 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:27.553831 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:27.553851 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:43:27.553917 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:27.553922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:27.554085 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:27.554227 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:27.554413 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.555416 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.556728 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.557433 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.558455 (+ 1022us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.558615 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:27.558662 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:43:27.558754 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1232,"thread_start_us":127,"threads_started":1}
I20260926 14:43:27.561208 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6c260390f740a78315fe7b21729502 (DEFAULT_TABLE table=test-table [id=578004b095e54a1791fabd969fe5a52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:27.561183 4214 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6c260390f740a78315fe7b21729502 (DEFAULT_TABLE table=test-table [id=578004b095e54a1791fabd969fe5a52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:27.561408 4479 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6c260390f740a78315fe7b21729502 (DEFAULT_TABLE table=test-table [id=578004b095e54a1791fabd969fe5a52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:43:27.562144 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6c260390f740a78315fe7b21729502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:27.562216 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6c260390f740a78315fe7b21729502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:27.562216 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6c260390f740a78315fe7b21729502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:27.568279 4563 tablet_bootstrap.cc:492] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c: Bootstrap starting.
I20260926 14:43:27.569643 4565 tablet_bootstrap.cc:492] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822: Bootstrap starting.
I20260926 14:43:27.570607 4563 tablet_bootstrap.cc:654] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:27.571458 4563 log.cc:824] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:27.571715 4565 tablet_bootstrap.cc:654] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:27.571991 4564 tablet_bootstrap.cc:492] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68: Bootstrap starting.
I20260926 14:43:27.572496 4565 log.cc:824] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:27.573808 4563 tablet_bootstrap.cc:492] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c: No bootstrap required, opened a new log
I20260926 14:43:27.573868 4564 tablet_bootstrap.cc:654] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:27.574163 4563 ts_tablet_manager.cc:1397] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:43:27.574424 4565 tablet_bootstrap.cc:492] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822: No bootstrap required, opened a new log
I20260926 14:43:27.574606 4565 ts_tablet_manager.cc:1397] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 14:43:27.574663 4564 log.cc:824] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:27.576175 4564 tablet_bootstrap.cc:492] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68: No bootstrap required, opened a new log
I20260926 14:43:27.576330 4564 ts_tablet_manager.cc:1397] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260926 14:43:27.578676 4565 raft_consensus.cc:348] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11213448e6104ad68db8ae9ad0353822" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39107 } } peers { permanent_uuid: "499830e7327e4a00888553bc4943de68" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 43941 } } peers { permanent_uuid: "41175da4854c497ba5629943428f9f1c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 46323 } }
I20260926 14:43:27.578676 4563 raft_consensus.cc:348] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11213448e6104ad68db8ae9ad0353822" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39107 } } peers { permanent_uuid: "499830e7327e4a00888553bc4943de68" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 43941 } } peers { permanent_uuid: "41175da4854c497ba5629943428f9f1c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 46323 } }
I20260926 14:43:27.578931 4563 raft_consensus.cc:374] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:27.578931 4565 raft_consensus.cc:374] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:27.578985 4563 raft_consensus.cc:729] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41175da4854c497ba5629943428f9f1c, State: Initialized, Role: FOLLOWER
I20260926 14:43:27.578992 4565 raft_consensus.cc:729] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11213448e6104ad68db8ae9ad0353822, State: Initialized, Role: FOLLOWER
I20260926 14:43:27.579217 4564 raft_consensus.cc:348] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11213448e6104ad68db8ae9ad0353822" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39107 } } peers { permanent_uuid: "499830e7327e4a00888553bc4943de68" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 43941 } } peers { permanent_uuid: "41175da4854c497ba5629943428f9f1c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 46323 } }
I20260926 14:43:27.579434 4564 raft_consensus.cc:374] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:27.579485 4564 raft_consensus.cc:729] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499830e7327e4a00888553bc4943de68, State: Initialized, Role: FOLLOWER
I20260926 14:43:27.579417 4565 consensus_queue.cc:260] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11213448e6104ad68db8ae9ad0353822" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39107 } } peers { permanent_uuid: "499830e7327e4a00888553bc4943de68" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 43941 } } peers { permanent_uuid: "41175da4854c497ba5629943428f9f1c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 46323 } }
I20260926 14:43:27.579417 4563 consensus_queue.cc:260] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11213448e6104ad68db8ae9ad0353822" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39107 } } peers { permanent_uuid: "499830e7327e4a00888553bc4943de68" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 43941 } } peers { permanent_uuid: "41175da4854c497ba5629943428f9f1c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 46323 } }
I20260926 14:43:27.579928 4564 consensus_queue.cc:260] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11213448e6104ad68db8ae9ad0353822" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39107 } } peers { permanent_uuid: "499830e7327e4a00888553bc4943de68" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 43941 } } peers { permanent_uuid: "41175da4854c497ba5629943428f9f1c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 46323 } }
I20260926 14:43:27.580379 4563 ts_tablet_manager.cc:1428] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260926 14:43:27.580379 4565 ts_tablet_manager.cc:1428] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260926 14:43:27.580731 4414 heartbeater.cc:503] Master 127.16.90.254:35747 was elected leader, sending a full tablet report...
I20260926 14:43:27.580731 4280 heartbeater.cc:503] Master 127.16.90.254:35747 was elected leader, sending a full tablet report...
I20260926 14:43:27.581835 4564 ts_tablet_manager.cc:1428] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:43:27.582779 4546 heartbeater.cc:503] Master 127.16.90.254:35747 was elected leader, sending a full tablet report...
W20260926 14:43:27.586968 4415 tablet.cc:2369] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:43:27.650871 4570 raft_consensus.cc:482] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:43:27.651072 4570 raft_consensus.cc:504] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11213448e6104ad68db8ae9ad0353822" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39107 } } peers { permanent_uuid: "499830e7327e4a00888553bc4943de68" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 43941 } } peers { permanent_uuid: "41175da4854c497ba5629943428f9f1c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 46323 } }
I20260926 14:43:27.652230 4570 leader_election.cc:288] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11213448e6104ad68db8ae9ad0353822 (127.16.90.193:39107), 499830e7327e4a00888553bc4943de68 (127.16.90.195:43941)
I20260926 14:43:27.655912 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.652436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:39107 (local address 127.16.90.194:41843)
0926 14:43:27.652593 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:27.652609 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:27.652714 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:27.652979 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:27.652981 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:27.652997 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:27.653117 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.653123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.653916 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.653919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.654795 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.654804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.655443 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.655447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.655556 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.655608 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 14:43:27.655698 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 14:43:27.655772 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":27}
I20260926 14:43:27.655928 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.652549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:36329 (local address 127.16.90.195:43941)
0926 14:43:27.652672 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:27.652676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:27.653048 (+ 372us) server_negotiation.cc:407] Connection header received
0926 14:43:27.653142 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:27.653145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:27.653210 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:27.653270 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:27.653523 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.654035 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.655024 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.655410 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.655517 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.655564 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:27.655769 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 14:43:27.655835 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 14:43:27.655916 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.652530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.90.194:41843 (local address 127.16.90.193:39107)
0926 14:43:27.652657 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:27.652662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:27.652679 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:43:27.652827 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:27.652830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:27.652882 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:27.652963 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:43:27.653219 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.653810 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.654910 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.655328 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.655419 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.655457 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:27.655718 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 14:43:27.655791 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:43:27.655943 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.652436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.195:43941 (local address 127.16.90.194:36329)
0926 14:43:27.652959 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:27.652973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:27.653063 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:27.653296 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:27.653299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:27.653315 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:43:27.653431 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.653437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.654138 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.654141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.654903 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:27.654912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.655532 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.655535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:27.655641 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.655683 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 14:43:27.655754 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:43:27.655813 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":163,"thread_start_us":139,"threads_started":1}
I20260926 14:43:27.656479 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6c260390f740a78315fe7b21729502" candidate_uuid: "41175da4854c497ba5629943428f9f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499830e7327e4a00888553bc4943de68" is_pre_election: true
I20260926 14:43:27.656577 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6c260390f740a78315fe7b21729502" candidate_uuid: "41175da4854c497ba5629943428f9f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11213448e6104ad68db8ae9ad0353822" is_pre_election: true
I20260926 14:43:27.656769 4499 raft_consensus.cc:2454] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41175da4854c497ba5629943428f9f1c in term 0.
I20260926 14:43:27.656843 4234 raft_consensus.cc:2454] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41175da4854c497ba5629943428f9f1c in term 0.
I20260926 14:43:27.657217 4302 leader_election.cc:302] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11213448e6104ad68db8ae9ad0353822, 41175da4854c497ba5629943428f9f1c, 499830e7327e4a00888553bc4943de68; no voters:
I20260926 14:43:27.657485 4570 raft_consensus.cc:2781] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:43:27.657555 4570 raft_consensus.cc:482] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:43:27.657596 4570 raft_consensus.cc:3037] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:27.658553 4570 raft_consensus.cc:504] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11213448e6104ad68db8ae9ad0353822" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39107 } } peers { permanent_uuid: "499830e7327e4a00888553bc4943de68" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 43941 } } peers { permanent_uuid: "41175da4854c497ba5629943428f9f1c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 46323 } }
I20260926 14:43:27.658869 4570 leader_election.cc:288] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [CANDIDATE]: Term 1 election: Requested vote from peers 11213448e6104ad68db8ae9ad0353822 (127.16.90.193:39107), 499830e7327e4a00888553bc4943de68 (127.16.90.195:43941)
I20260926 14:43:27.659170 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6c260390f740a78315fe7b21729502" candidate_uuid: "41175da4854c497ba5629943428f9f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11213448e6104ad68db8ae9ad0353822"
I20260926 14:43:27.659276 4234 raft_consensus.cc:3037] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:27.659792 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6c260390f740a78315fe7b21729502" candidate_uuid: "41175da4854c497ba5629943428f9f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499830e7327e4a00888553bc4943de68"
I20260926 14:43:27.659894 4499 raft_consensus.cc:3037] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:27.660346 4234 raft_consensus.cc:2454] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41175da4854c497ba5629943428f9f1c in term 1.
I20260926 14:43:27.660624 4300 leader_election.cc:302] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11213448e6104ad68db8ae9ad0353822, 41175da4854c497ba5629943428f9f1c; no voters:
I20260926 14:43:27.660951 4499 raft_consensus.cc:2454] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41175da4854c497ba5629943428f9f1c in term 1.
I20260926 14:43:27.661136 4570 raft_consensus.cc:2781] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:27.661353 4570 raft_consensus.cc:686] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 1 LEADER]: Becoming Leader. State: Replica: 41175da4854c497ba5629943428f9f1c, State: Running, Role: LEADER
I20260926 14:43:27.661654 4570 consensus_queue.cc:237] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11213448e6104ad68db8ae9ad0353822" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39107 } } peers { permanent_uuid: "499830e7327e4a00888553bc4943de68" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 43941 } } peers { permanent_uuid: "41175da4854c497ba5629943428f9f1c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 46323 } }
I20260926 14:43:27.664614 4091 catalog_manager.cc:5495] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 41175da4854c497ba5629943428f9f1c (127.16.90.194). New cstate: current_term: 1 leader_uuid: "41175da4854c497ba5629943428f9f1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11213448e6104ad68db8ae9ad0353822" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "499830e7327e4a00888553bc4943de68" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 43941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41175da4854c497ba5629943428f9f1c" member_type: VOTER last_known_addr { host: "127.16.90.194" port: 46323 } health_report { overall_health: HEALTHY } } }
W20260926 14:43:27.669442 4281 tablet.cc:2369] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:43:27.676225 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.673300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.16.90.194:46323)
0926 14:43:27.673397 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:27.673400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:27.673413 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:43:27.673499 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:27.673501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:27.673546 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:27.673635 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:27.673818 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.674284 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.675047 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:27.675245 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:27.675308 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:27.675517 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:27.675943 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:27.676029 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:27.676069 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 14:43:27.676110 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 14:43:27.681725 4234 raft_consensus.cc:1261] T 1d6c260390f740a78315fe7b21729502 P 11213448e6104ad68db8ae9ad0353822 [term 1 FOLLOWER]: Refusing update from remote peer 41175da4854c497ba5629943428f9f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:27.681726 4499 raft_consensus.cc:1261] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68 [term 1 FOLLOWER]: Refusing update from remote peer 41175da4854c497ba5629943428f9f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:27.682322 4570 consensus_queue.cc:1035] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11213448e6104ad68db8ae9ad0353822" member_type: VOTER last_known_addr { host: "127.16.90.193" port: 39107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:27.682493 4573 consensus_queue.cc:1035] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "499830e7327e4a00888553bc4943de68" member_type: VOTER last_known_addr { host: "127.16.90.195" port: 43941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:27.691145 4577 mvcc.cc:205] Tried to move back new op lower bound from 7333616876257353728 to 7333616876184182784. Current Snapshot: MvccSnapshot[applied={T|T < 7333616876257353728}]
I20260926 14:43:27.697054 4091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:43:27.697201 4091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:43:27.699357 4091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49e95646655f403e81b5599e051795b9: Sending DeleteTablet for 3 replicas of tablet 1d6c260390f740a78315fe7b21729502
I20260926 14:43:27.700148 4479 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6c260390f740a78315fe7b21729502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:27 UTC) from {username='slave'} at 127.0.0.1:45332
I20260926 14:43:27.700304 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6c260390f740a78315fe7b21729502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:27 UTC) from {username='slave'} at 127.0.0.1:44082
I20260926 14:43:27.700309 4214 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6c260390f740a78315fe7b21729502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:43:27 UTC) from {username='slave'} at 127.0.0.1:60214
I20260926 14:43:27.700810 4587 tablet_replica.cc:330] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c: stopping tablet replica
I20260926 14:43:27.700953 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 4152
I20260926 14:43:27.701776 4587 raft_consensus.cc:2229] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:43:27.702116 4587 raft_consensus.cc:2258] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:43:27.702417 4586 tablet_replica.cc:330] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68: stopping tablet replica
I20260926 14:43:27.702704 4586 raft_consensus.cc:2229] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:43:27.702982 4586 raft_consensus.cc:2258] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:43:27.703588 4587 ts_tablet_manager.cc:1905] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:43:27.704334 4586 ts_tablet_manager.cc:1905] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:43:27.706286 4587 ts_tablet_manager.cc:1918] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:43:27.706374 4587 log.cc:1196] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-1/wal/wals/1d6c260390f740a78315fe7b21729502
I20260926 14:43:27.706661 4587 ts_tablet_manager.cc:1939] T 1d6c260390f740a78315fe7b21729502 P 41175da4854c497ba5629943428f9f1c: Deleting consensus metadata
I20260926 14:43:27.706758 4586 ts_tablet_manager.cc:1918] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:43:27.706826 4586 log.cc:1196] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68: Deleting WAL directory at /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433562403030-16747-0/minicluster-data/ts-2/wal/wals/1d6c260390f740a78315fe7b21729502
I20260926 14:43:27.707064 4586 ts_tablet_manager.cc:1939] T 1d6c260390f740a78315fe7b21729502 P 499830e7327e4a00888553bc4943de68: Deleting consensus metadata
I20260926 14:43:27.707837 4076 catalog_manager.cc:4841] TS 41175da4854c497ba5629943428f9f1c (127.16.90.194:46323): tablet 1d6c260390f740a78315fe7b21729502 (table test-table [id=578004b095e54a1791fabd969fe5a52c]) successfully deleted
I20260926 14:43:27.708072 4078 catalog_manager.cc:4841] TS 499830e7327e4a00888553bc4943de68 (127.16.90.195:43941): tablet 1d6c260390f740a78315fe7b21729502 (table test-table [id=578004b095e54a1791fabd969fe5a52c]) successfully deleted
W20260926 14:43:27.709383 4076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.90.193:39107 (error 108)
I20260926 14:43:27.710269 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:27.710027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.193:39107 (local address 127.0.0.1:60216)
0926 14:43:27.710138 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:27.710199 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.193:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:43:27.710392 4076 catalog_manager.cc:4584] TS 11213448e6104ad68db8ae9ad0353822 (127.16.90.193:39107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d6c260390f740a78315fe7b21729502: Network error: Client connection negotiation failed: client connection to 127.16.90.193:39107: connect: Connection refused (error 111)
I20260926 14:43:27.710713 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 4284
I20260926 14:43:27.715685 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 4417
I20260926 14:43:27.720281 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 4060
2026-09-26T14:43:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1000 ms)
[----------] 6 tests from EncryptionPolicyTest (9894 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:27 dist-test-slave-x4sn krb5kdc[4592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:27 dist-test-slave-x4sn krb5kdc[4592](info): set up 2 sockets
Sep 26 14:43:27 dist-test-slave-x4sn krb5kdc[4592](info): commencing operation
krb5kdc: starting...
W20260926 14:43:28.669062 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:28 dist-test-slave-x4sn krb5kdc[4592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:43:28.783779 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:28.889391 4605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:28.889649 4605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:28.889693 4605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:28.893754 4605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:28.893806 4605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:28.893823 4605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:28.893838 4605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:28.893852 4605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:28.899001 4605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41333
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:28.899963 4605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:28.900870 4605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:28.908689 4613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:28.908713 4611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:28.908689 4610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:28.909030 4605 server_base.cc:1034] running on GCE node
I20260926 14:43:28.909574 4605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:28.910645 4605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:28.911792 4605 hybrid_clock.cc:648] HybridClock initialized: now 1790433808911772 us; error 44 us; skew 500 ppm
Sep 26 14:43:28 dist-test-slave-x4sn krb5kdc[4592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:28.914377 4605 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:43:28.915218 4605 webserver.cc:460] Webserver started at http://127.0.0.1:42301/ using document root <none> and password file <none>
I20260926 14:43:28.915777 4605 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:28.915848 4605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:28.916043 4605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:28.918063 4605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "9628c4105cff4c269ae9ec7316239e7d"
format_stamp: "Formatted at 2026-09-26 14:43:28 on dist-test-slave-x4sn"
server_key: "b3b347491e23b345d54598f7c4605d2d"
server_key_iv: "e666ede86dd07adf391b6f17c8b83ae2"
server_key_version: "encryptionkey@0"
I20260926 14:43:28.918627 4605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "9628c4105cff4c269ae9ec7316239e7d"
format_stamp: "Formatted at 2026-09-26 14:43:28 on dist-test-slave-x4sn"
server_key: "b3b347491e23b345d54598f7c4605d2d"
server_key_iv: "e666ede86dd07adf391b6f17c8b83ae2"
server_key_version: "encryptionkey@0"
I20260926 14:43:28.922235 4605 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:43:28.924518 4620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:28.925688 4605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:28.925810 4605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "9628c4105cff4c269ae9ec7316239e7d"
format_stamp: "Formatted at 2026-09-26 14:43:28 on dist-test-slave-x4sn"
server_key: "b3b347491e23b345d54598f7c4605d2d"
server_key_iv: "e666ede86dd07adf391b6f17c8b83ae2"
server_key_version: "encryptionkey@0"
I20260926 14:43:28.925906 4605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:28.946591 4605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:28.949806 4605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:28.949993 4605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:28.956697 4605 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41333
I20260926 14:43:28.956727 4672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41333 every 8 connection(s)
I20260926 14:43:28.957665 4605 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:28.959347 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 4605
I20260926 14:43:28.959467 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:28.959785 16747 external_mini_cluster.cc:1428] Setting key 99996d633409996fff6fb2ddee4a7707
I20260926 14:43:28.960635 4673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:28.967108 4673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d: Bootstrap starting.
Sep 26 14:43:28 dist-test-slave-x4sn krb5kdc[4592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:43:28.969515 4673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:28.970293 4673 log.cc:824] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:28.972142 4673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d: No bootstrap required, opened a new log
I20260926 14:43:28.974179 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:28.961162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47201 (local address 127.0.0.1:41333)
0926 14:43:28.961580 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:28.961601 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:28.961643 (+ 42us) server_negotiation.cc:407] Connection header received
0926 14:43:28.962473 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:28.962496 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:28.962934 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:28.963309 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:28.964199 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:28.965631 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:28.966522 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:28.967191 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:28.967346 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:28.969931 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:28.969936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:28.969948 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:28.969973 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:28.971468 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:28.972007 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:28.972012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:28.972016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:28.972121 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:28.972448 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:28.972450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:28.972452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:28.972818 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:28.972966 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:28.973267 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 14:43:28.973476 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260926 14:43:28.975710 4673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9628c4105cff4c269ae9ec7316239e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41333 } }
I20260926 14:43:28.975929 4673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:28.975991 4673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9628c4105cff4c269ae9ec7316239e7d, State: Initialized, Role: FOLLOWER
I20260926 14:43:28.976372 4673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9628c4105cff4c269ae9ec7316239e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41333 } }
I20260926 14:43:28.976486 4673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:28.976541 4673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:28.976629 4673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:28.976832 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:28.977680 4673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9628c4105cff4c269ae9ec7316239e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41333 } }
I20260926 14:43:28.978030 4673 leader_election.cc:302] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9628c4105cff4c269ae9ec7316239e7d; no voters:
I20260926 14:43:28.978363 4673 leader_election.cc:288] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:28.978440 4683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:28.978636 4683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [term 1 LEADER]: Becoming Leader. State: Replica: 9628c4105cff4c269ae9ec7316239e7d, State: Running, Role: LEADER
I20260926 14:43:28.979372 4683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9628c4105cff4c269ae9ec7316239e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41333 } }
I20260926 14:43:28.979602 4673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:28.980974 4685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9628c4105cff4c269ae9ec7316239e7d. Latest consensus state: current_term: 1 leader_uuid: "9628c4105cff4c269ae9ec7316239e7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9628c4105cff4c269ae9ec7316239e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41333 } } }
I20260926 14:43:28.981096 4685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:28.981382 4684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9628c4105cff4c269ae9ec7316239e7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9628c4105cff4c269ae9ec7316239e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41333 } } }
I20260926 14:43:28.981796 4684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d [sys.catalog]: This master's current role is: LEADER
W20260926 14:43:28.984046 4699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:43:28.984122 4699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:43:28.984195 4698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:28.985028 4698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:28.987810 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:28.978912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51135 (local address 127.0.0.1:41333)
0926 14:43:28.979025 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:28.979029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:28.979223 (+ 194us) server_negotiation.cc:407] Connection header received
0926 14:43:28.979396 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:28.979399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:28.979444 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:28.979534 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:28.980471 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:28.981237 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:28.982422 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:28.982753 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:28.982845 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:28.984511 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:28.984515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:28.984517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:28.984541 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:28.986100 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:28.986625 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:28.986628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:28.986630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:28.986715 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:28.987017 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:28.987020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:28.987022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:28.987200 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:28.987322 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:28.987570 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 14:43:28.987651 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:43:28.990653 4698 catalog_manager.cc:1347] Generated new cluster ID: 7d170482d5a84083a029e8104a343fb9
I20260926 14:43:28.990713 4698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:29.003032 4698 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:29.004292 4698 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:29.014055 4698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9628c4105cff4c269ae9ec7316239e7d: Generated new TSK 0
I20260926 14:43:29.014735 4698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:29.029361 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 4605
W20260926 14:43:29.034578 4590 connection.cc:535] client connection to 127.0.0.1:41333 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41333): Connection reset by peer (error 104)
2026-09-26T14:43:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:29 dist-test-slave-x4sn krb5kdc[4705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:29 dist-test-slave-x4sn krb5kdc[4705](info): set up 2 sockets
Sep 26 14:43:29 dist-test-slave-x4sn krb5kdc[4705](info): commencing operation
krb5kdc: starting...
W20260926 14:43:29.985167 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:30 dist-test-slave-x4sn krb5kdc[4705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:43:30.098356 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:30.208730 4718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:30.208999 4718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:30.209039 4718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:30.213035 4718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:30.213094 4718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:30.213112 4718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:30.213132 4718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:30.213150 4718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:30.218272 4718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33991
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:30.219241 4718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:30.220248 4718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:30.228240 4723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:30.228354 4718 server_base.cc:1034] running on GCE node
W20260926 14:43:30.228242 4724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:30.228242 4726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:30.229041 4718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:30.230108 4718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:30.231262 4718 hybrid_clock.cc:648] HybridClock initialized: now 1790433810231240 us; error 33 us; skew 500 ppm
Sep 26 14:43:30 dist-test-slave-x4sn krb5kdc[4705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:30.238443 4718 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:43:30.239328 4718 webserver.cc:460] Webserver started at http://127.0.0.1:44033/ using document root <none> and password file <none>
I20260926 14:43:30.239907 4718 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:30.239974 4718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:30.240201 4718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:30.242166 4718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "33f9178a388e475b844a677a9180e46d"
format_stamp: "Formatted at 2026-09-26 14:43:30 on dist-test-slave-x4sn"
server_key: "3018d696523af73c792f6a2f03a79e89"
server_key_iv: "a24016b79c85504debb92023772e33db"
server_key_version: "encryptionkey@0"
I20260926 14:43:30.242672 4718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "33f9178a388e475b844a677a9180e46d"
format_stamp: "Formatted at 2026-09-26 14:43:30 on dist-test-slave-x4sn"
server_key: "3018d696523af73c792f6a2f03a79e89"
server_key_iv: "a24016b79c85504debb92023772e33db"
server_key_version: "encryptionkey@0"
I20260926 14:43:30.246151 4718 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 14:43:30.248391 4733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:30.249614 4718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:43:30.249727 4718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "33f9178a388e475b844a677a9180e46d"
format_stamp: "Formatted at 2026-09-26 14:43:30 on dist-test-slave-x4sn"
server_key: "3018d696523af73c792f6a2f03a79e89"
server_key_iv: "a24016b79c85504debb92023772e33db"
server_key_version: "encryptionkey@0"
I20260926 14:43:30.249814 4718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:30.261981 4718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:30.264720 4718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:30.264886 4718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:30.271391 4718 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33991
I20260926 14:43:30.271417 4785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33991 every 8 connection(s)
I20260926 14:43:30.272317 4718 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:30.273524 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 4718
I20260926 14:43:30.273643 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:30.273904 16747 external_mini_cluster.cc:1428] Setting key 1a32fcbc7810dd1653054005298db4a3
I20260926 14:43:30.275043 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:30.280380 4786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d: Bootstrap starting.
I20260926 14:43:30.282168 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:30.275159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45533 (local address 127.0.0.1:33991)
0926 14:43:30.275573 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:30.275594 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:30.275629 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:43:30.276294 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:30.276311 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:30.276640 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:30.276928 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:30.277315 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:30.278270 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:30.279134 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:30.279605 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:30.279784 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:30.280618 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:30.280622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:30.280634 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:30.280985 (+ 351us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:30.281036 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:30.281057 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:30.281167 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:30.281345 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 14:43:30.281468 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260926 14:43:30.282514 4786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:30.283247 4786 log.cc:824] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:30.284890 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:30.285211 4786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d: No bootstrap required, opened a new log
I20260926 14:43:30.288677 4786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f9178a388e475b844a677a9180e46d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33991 } }
I20260926 14:43:30.288885 4786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:30.288937 4786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f9178a388e475b844a677a9180e46d, State: Initialized, Role: FOLLOWER
I20260926 14:43:30.289331 4786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f9178a388e475b844a677a9180e46d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33991 } }
I20260926 14:43:30.289484 4786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:30.289600 4786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:30.289709 4786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:30.290271 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:30.286684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44881 (local address 127.0.0.1:33991)
0926 14:43:30.286779 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:30.286783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:30.286955 (+ 172us) server_negotiation.cc:407] Connection header received
0926 14:43:30.287062 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:30.287065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:30.287121 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:30.287182 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:30.287420 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:30.287932 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:30.288698 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:30.288906 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:30.288988 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:30.289867 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:30.289870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:30.289873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:30.289885 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:30.289911 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:30.289924 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:30.290017 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:30.290097 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 14:43:30.290129 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:43:30.290678 4786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f9178a388e475b844a677a9180e46d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33991 } }
I20260926 14:43:30.291015 4786 leader_election.cc:302] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33f9178a388e475b844a677a9180e46d; no voters:
I20260926 14:43:30.291347 4786 leader_election.cc:288] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:30.291409 4796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:30.291625 4796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [term 1 LEADER]: Becoming Leader. State: Replica: 33f9178a388e475b844a677a9180e46d, State: Running, Role: LEADER
I20260926 14:43:30.291954 4796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f9178a388e475b844a677a9180e46d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33991 } }
I20260926 14:43:30.292433 4786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:30.293432 4797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33f9178a388e475b844a677a9180e46d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f9178a388e475b844a677a9180e46d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33991 } } }
I20260926 14:43:30.293551 4797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:30.293818 4798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33f9178a388e475b844a677a9180e46d. Latest consensus state: current_term: 1 leader_uuid: "33f9178a388e475b844a677a9180e46d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f9178a388e475b844a677a9180e46d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33991 } } }
I20260926 14:43:30.293905 4798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:30.294015 4805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:30.296864 4805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:30.301676 4805 catalog_manager.cc:1347] Generated new cluster ID: aa8a5092d91b49b494792d9803dde322
I20260926 14:43:30.301751 4805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:30.332712 4805 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:30.333984 4805 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:30.344219 4805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33f9178a388e475b844a677a9180e46d: Generated new TSK 0
I20260926 14:43:30.345049 4805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:30.352891 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 4718
W20260926 14:43:30.357745 4703 connection.cc:535] client connection to 127.0.0.1:33991 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33991): Connection reset by peer (error 104)
2026-09-26T14:43:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:30 dist-test-slave-x4sn krb5kdc[4817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:30 dist-test-slave-x4sn krb5kdc[4817](info): set up 2 sockets
Sep 26 14:43:30 dist-test-slave-x4sn krb5kdc[4817](info): commencing operation
krb5kdc: starting...
W20260926 14:43:31.301222 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:31 dist-test-slave-x4sn krb5kdc[4817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:43:31.412499 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:31.518502 4830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:31.518766 4830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:31.518810 4830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:31.522790 4830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:31.522853 4830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:31.522871 4830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:31.522890 4830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:31.522918 4830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:31.528512 4830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:31.529564 4830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:31.530582 4830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:31.538878 4836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:31.538878 4835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:31.538995 4838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:31.539155 4830 server_base.cc:1034] running on GCE node
I20260926 14:43:31.539733 4830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:31.540834 4830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:31.541980 4830 hybrid_clock.cc:648] HybridClock initialized: now 1790433811541957 us; error 37 us; skew 500 ppm
Sep 26 14:43:31 dist-test-slave-x4sn krb5kdc[4817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:31.544581 4830 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:43:31.545475 4830 webserver.cc:460] Webserver started at http://127.0.0.1:45733/ using document root <none> and password file <none>
I20260926 14:43:31.546006 4830 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:31.546072 4830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:31.546259 4830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:31.548283 4830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "932b2f7cba604901a17e2df6d879c6ec"
format_stamp: "Formatted at 2026-09-26 14:43:31 on dist-test-slave-x4sn"
server_key: "cccb26fa54c16765b31886775251bdb6"
server_key_iv: "a6874b1786a49d97f8f13db335d6680f"
server_key_version: "encryptionkey@0"
I20260926 14:43:31.548805 4830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "932b2f7cba604901a17e2df6d879c6ec"
format_stamp: "Formatted at 2026-09-26 14:43:31 on dist-test-slave-x4sn"
server_key: "cccb26fa54c16765b31886775251bdb6"
server_key_iv: "a6874b1786a49d97f8f13db335d6680f"
server_key_version: "encryptionkey@0"
I20260926 14:43:31.552313 4830 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:43:31.554569 4845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:31.555766 4830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:31.555873 4830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "932b2f7cba604901a17e2df6d879c6ec"
format_stamp: "Formatted at 2026-09-26 14:43:31 on dist-test-slave-x4sn"
server_key: "cccb26fa54c16765b31886775251bdb6"
server_key_iv: "a6874b1786a49d97f8f13db335d6680f"
server_key_version: "encryptionkey@0"
I20260926 14:43:31.555960 4830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:31.560328 4830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:31.562904 4830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:31.563056 4830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:31.569559 4830 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40153
I20260926 14:43:31.569582 4897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40153 every 8 connection(s)
I20260926 14:43:31.570415 4830 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:31.573083 4898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:31.577912 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 4830
I20260926 14:43:31.578047 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:31.578369 16747 external_mini_cluster.cc:1428] Setting key e6e10cd07eeb4d4f9932ac5d787b979c
I20260926 14:43:31.578511 4898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec: Bootstrap starting.
I20260926 14:43:31.580948 4898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:31.581681 4898 log.cc:824] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:31.583261 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:31.579586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58613 (local address 127.0.0.1:40153)
0926 14:43:31.580045 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:31.580059 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:31.580094 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:43:31.580711 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:31.580728 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:31.580993 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:31.581312 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:31.582076 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:31.582081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:31.582097 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:31.582109 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:31.582185 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:31.582216 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:31.582323 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:31.582442 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 14:43:31.582576 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":130,"threads_started":1}
I20260926 14:43:31.583586 4898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec: No bootstrap required, opened a new log
I20260926 14:43:31.585712 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:31.587356 4898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932b2f7cba604901a17e2df6d879c6ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40153 } }
I20260926 14:43:31.587616 4898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:31.587695 4898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932b2f7cba604901a17e2df6d879c6ec, State: Initialized, Role: FOLLOWER
I20260926 14:43:31.588135 4898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932b2f7cba604901a17e2df6d879c6ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40153 } }
I20260926 14:43:31.588268 4898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:31.588326 4898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:31.588421 4898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:31.589504 4898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932b2f7cba604901a17e2df6d879c6ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40153 } }
I20260926 14:43:31.589624 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:31.587982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60833 (local address 127.0.0.1:40153)
0926 14:43:31.588093 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:31.588098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:31.588308 (+ 210us) server_negotiation.cc:407] Connection header received
0926 14:43:31.588427 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:31.588430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:31.588466 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:31.588550 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:31.589271 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:31.589274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:31.589276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:31.589285 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:31.589308 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:31.589318 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:31.589377 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:31.589478 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 14:43:31.589504 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:43:31.589888 4898 leader_election.cc:302] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 932b2f7cba604901a17e2df6d879c6ec; no voters:
I20260926 14:43:31.590188 4898 leader_election.cc:288] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:31.590268 4908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:31.590466 4908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [term 1 LEADER]: Becoming Leader. State: Replica: 932b2f7cba604901a17e2df6d879c6ec, State: Running, Role: LEADER
I20260926 14:43:31.590796 4908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932b2f7cba604901a17e2df6d879c6ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40153 } }
I20260926 14:43:31.591418 4898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:31.592363 4910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932b2f7cba604901a17e2df6d879c6ec. Latest consensus state: current_term: 1 leader_uuid: "932b2f7cba604901a17e2df6d879c6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932b2f7cba604901a17e2df6d879c6ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40153 } } }
I20260926 14:43:31.592478 4910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:31.592864 4917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:31.592835 4909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "932b2f7cba604901a17e2df6d879c6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932b2f7cba604901a17e2df6d879c6ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40153 } } }
I20260926 14:43:31.593029 4909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:31.595919 4917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:31.601094 4917 catalog_manager.cc:1347] Generated new cluster ID: 5aac4a8c75db4463b055a930a6605fd9
I20260926 14:43:31.601301 4917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:31.616870 4917 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:31.617172 4917 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:31.627650 4917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 932b2f7cba604901a17e2df6d879c6ec: Generated new TSK 0
I20260926 14:43:31.628324 4917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:31.636274 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 4830
2026-09-26T14:43:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:31 dist-test-slave-x4sn krb5kdc[4929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:31 dist-test-slave-x4sn krb5kdc[4929](info): set up 2 sockets
Sep 26 14:43:31 dist-test-slave-x4sn krb5kdc[4929](info): commencing operation
krb5kdc: starting...
W20260926 14:43:32.581094 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:32 dist-test-slave-x4sn krb5kdc[4929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:43:32.692416 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:32.800001 4942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:32.800338 4942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:32.800405 4942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:32.804386 4942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:32.804440 4942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:32.804457 4942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:32.804474 4942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:32.804502 4942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:32.809612 4942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:32.810561 4942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:32.811509 4942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:43:32.819573 4942 server_base.cc:1034] running on GCE node
W20260926 14:43:32.819530 4950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:32.819537 4948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:32.819537 4947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:32.820330 4942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:32.821342 4942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:32.822520 4942 hybrid_clock.cc:648] HybridClock initialized: now 1790433812822494 us; error 36 us; skew 500 ppm
Sep 26 14:43:32 dist-test-slave-x4sn krb5kdc[4929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:32.825205 4942 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:43:32.826067 4942 webserver.cc:460] Webserver started at http://127.0.0.1:46187/ using document root <none> and password file <none>
I20260926 14:43:32.826583 4942 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:32.826643 4942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:32.826838 4942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:32.828856 4942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "ef9840a631b74ae3b175a266d49cb5fb"
format_stamp: "Formatted at 2026-09-26 14:43:32 on dist-test-slave-x4sn"
server_key: "5567f3103c896f30f05fb652dd97e0a4"
server_key_iv: "a3fc79960c13a02f6dd35d066526bcc2"
server_key_version: "encryptionkey@0"
I20260926 14:43:32.829355 4942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "ef9840a631b74ae3b175a266d49cb5fb"
format_stamp: "Formatted at 2026-09-26 14:43:32 on dist-test-slave-x4sn"
server_key: "5567f3103c896f30f05fb652dd97e0a4"
server_key_iv: "a3fc79960c13a02f6dd35d066526bcc2"
server_key_version: "encryptionkey@0"
I20260926 14:43:32.832814 4942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:43:32.835001 4957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:32.836166 4942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:32.836277 4942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "ef9840a631b74ae3b175a266d49cb5fb"
format_stamp: "Formatted at 2026-09-26 14:43:32 on dist-test-slave-x4sn"
server_key: "5567f3103c896f30f05fb652dd97e0a4"
server_key_iv: "a3fc79960c13a02f6dd35d066526bcc2"
server_key_version: "encryptionkey@0"
I20260926 14:43:32.836362 4942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:32.848145 4942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:32.850646 4942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:32.850808 4942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:32.857690 4942 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38299
I20260926 14:43:32.857707 5009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38299 every 8 connection(s)
I20260926 14:43:32.858613 4942 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:32.861289 5010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:32.866923 5010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb: Bootstrap starting.
I20260926 14:43:32.868144 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 4942
I20260926 14:43:32.868266 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:32.868498 16747 external_mini_cluster.cc:1428] Setting key 7f4dd93a16a3451ada759c78f7bdca8e
I20260926 14:43:32.869439 5010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:32.870232 5010 log.cc:824] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:32.872238 5010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb: No bootstrap required, opened a new log
I20260926 14:43:32.876371 5010 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9840a631b74ae3b175a266d49cb5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38299 } }
I20260926 14:43:32.876652 5010 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:32.876722 5010 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9840a631b74ae3b175a266d49cb5fb, State: Initialized, Role: FOLLOWER
Sep 26 14:43:32 dist-test-slave-x4sn krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:43:32.877125 5010 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9840a631b74ae3b175a266d49cb5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38299 } }
I20260926 14:43:32.877277 5010 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:32.877336 5010 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:32.877430 5010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:32.878484 5010 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9840a631b74ae3b175a266d49cb5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38299 } }
I20260926 14:43:32.878857 5010 leader_election.cc:302] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef9840a631b74ae3b175a266d49cb5fb; no voters:
I20260926 14:43:32.879195 5010 leader_election.cc:288] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:32.879287 5015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:32.879490 5015 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [term 1 LEADER]: Becoming Leader. State: Replica: ef9840a631b74ae3b175a266d49cb5fb, State: Running, Role: LEADER
I20260926 14:43:32.879860 5015 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9840a631b74ae3b175a266d49cb5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38299 } }
I20260926 14:43:32.880647 5010 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:32.881565 5017 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9840a631b74ae3b175a266d49cb5fb. Latest consensus state: current_term: 1 leader_uuid: "ef9840a631b74ae3b175a266d49cb5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9840a631b74ae3b175a266d49cb5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38299 } } }
I20260926 14:43:32.881567 5016 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef9840a631b74ae3b175a266d49cb5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9840a631b74ae3b175a266d49cb5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38299 } } }
I20260926 14:43:32.881722 5017 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:32.881722 5016 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:32.883495 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:32.869487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42625 (local address 127.0.0.1:38299)
0926 14:43:32.869932 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:32.869947 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:32.869983 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:43:32.870628 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:32.870647 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:32.870968 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:32.871240 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:32.872112 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:32.873110 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:32.874366 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:32.874843 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:32.874957 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:32.877738 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:32.877744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:32.877760 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:32.877789 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:32.879701 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:32.880247 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:32.880253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:32.880258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:32.880400 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:32.880740 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:32.880743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:32.880745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:32.881258 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:32.882044 (+ 786us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:32.882208 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 14:43:32.882522 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":142,"threads_started":1}
W20260926 14:43:32.885371 5030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:43:32.885432 5030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:43:32.885488 5024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:32.886161 5024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:32.888666 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:32.891571 5024 catalog_manager.cc:1347] Generated new cluster ID: afb2fa48175a41f682c6acd9b60264ef
I20260926 14:43:32.891623 5024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:32.899860 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:32.890724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42143 (local address 127.0.0.1:38299)
0926 14:43:32.890837 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:32.890841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:32.891992 (+ 1151us) server_negotiation.cc:407] Connection header received
0926 14:43:32.892121 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:32.892124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:32.892172 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:32.892247 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:32.892993 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:32.893757 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:32.894844 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:32.895175 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:32.895256 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:32.896768 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:32.896771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:32.896774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:32.896797 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:32.898305 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:32.898727 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:32.898730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:32.898732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:32.898801 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:32.899024 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:32.899027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:32.899028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:32.899211 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:32.899333 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:32.899567 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 14:43:32.899707 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:43:32.902292 5024 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:32.903407 5024 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:32.908667 5024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef9840a631b74ae3b175a266d49cb5fb: Generated new TSK 0
I20260926 14:43:32.909283 5024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:32.914376 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 4942
2026-09-26T14:43:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:32 dist-test-slave-x4sn krb5kdc[5041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:32 dist-test-slave-x4sn krb5kdc[5041](info): set up 2 sockets
Sep 26 14:43:32 dist-test-slave-x4sn krb5kdc[5041](info): commencing operation
krb5kdc: starting...
W20260926 14:43:33.861143 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:33 dist-test-slave-x4sn krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:43:33.974681 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:34.079572 5054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:34.079849 5054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:34.079891 5054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:34.083855 5054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:34.083909 5054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:34.083926 5054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:34.083940 5054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:34.083954 5054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:34.089040 5054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:34.090030 5054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:34.090994 5054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:34.098924 5059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:34.099004 5060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:34.098924 5062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:34.098985 5054 server_base.cc:1034] running on GCE node
I20260926 14:43:34.099648 5054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:34.100714 5054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:34.101866 5054 hybrid_clock.cc:648] HybridClock initialized: now 1790433814101852 us; error 30 us; skew 500 ppm
Sep 26 14:43:34 dist-test-slave-x4sn krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:34.104421 5054 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:43:34.105216 5054 webserver.cc:460] Webserver started at http://127.0.0.1:46179/ using document root <none> and password file <none>
I20260926 14:43:34.105711 5054 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:34.105777 5054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:34.105971 5054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:34.107975 5054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "75fc00515d0a41a9a2d7e0c16d810921"
format_stamp: "Formatted at 2026-09-26 14:43:34 on dist-test-slave-x4sn"
server_key: "4145335301434a87ce099b8bdf7d56ba"
server_key_iv: "b4de85d1b55b4b77db4275367442fff4"
server_key_version: "encryptionkey@0"
I20260926 14:43:34.108497 5054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "75fc00515d0a41a9a2d7e0c16d810921"
format_stamp: "Formatted at 2026-09-26 14:43:34 on dist-test-slave-x4sn"
server_key: "4145335301434a87ce099b8bdf7d56ba"
server_key_iv: "b4de85d1b55b4b77db4275367442fff4"
server_key_version: "encryptionkey@0"
I20260926 14:43:34.112008 5054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:43:34.114204 5069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:34.115337 5054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:34.115442 5054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "75fc00515d0a41a9a2d7e0c16d810921"
format_stamp: "Formatted at 2026-09-26 14:43:34 on dist-test-slave-x4sn"
server_key: "4145335301434a87ce099b8bdf7d56ba"
server_key_iv: "b4de85d1b55b4b77db4275367442fff4"
server_key_version: "encryptionkey@0"
I20260926 14:43:34.115525 5054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:34.137107 5054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:34.139833 5054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:34.140019 5054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:34.146636 5054 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38161
I20260926 14:43:34.146653 5121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38161 every 8 connection(s)
I20260926 14:43:34.147533 5054 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:34.150276 5122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:34.150290 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 5054
I20260926 14:43:34.150400 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:34.150642 16747 external_mini_cluster.cc:1428] Setting key 6b6f19792b6960ade423b1a1f5577c90
I20260926 14:43:34.157658 5122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921: Bootstrap starting.
I20260926 14:43:34.160468 5122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:34.161168 5125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:34.152030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57241 (local address 127.0.0.1:38161)
0926 14:43:34.152538 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:34.152558 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:34.152606 (+ 48us) server_negotiation.cc:407] Connection header received
0926 14:43:34.153726 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:34.153751 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:34.154219 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:34.154616 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:34.155031 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:34.156570 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:34.157453 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:34.158146 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:34.158312 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:34.159082 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:34.159089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:34.159105 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:34.159602 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:34.159704 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:34.159741 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:34.159846 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:34.159929 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 14:43:34.160162 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":167,"threads_started":1}
I20260926 14:43:34.161372 5122 log.cc:824] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:34.163583 5122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921: No bootstrap required, opened a new log
I20260926 14:43:34.164093 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:34.167658 5122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fc00515d0a41a9a2d7e0c16d810921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } }
I20260926 14:43:34.167929 5122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:34.167984 5122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75fc00515d0a41a9a2d7e0c16d810921, State: Initialized, Role: FOLLOWER
I20260926 14:43:34.168457 5122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fc00515d0a41a9a2d7e0c16d810921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } }
I20260926 14:43:34.168629 5122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:34.168710 5122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:34.168818 5122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:34.169974 5125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:34.165922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40857 (local address 127.0.0.1:38161)
0926 14:43:34.166066 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:34.166071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:34.166147 (+ 76us) server_negotiation.cc:407] Connection header received
0926 14:43:34.166240 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:34.166244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:34.166294 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:34.166392 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:34.166577 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:34.167338 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:34.168222 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:34.168589 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:34.168709 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:34.169473 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:34.169478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:34.169480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:34.169494 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:34.169526 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:34.169541 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:34.169626 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:34.169759 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 14:43:34.169818 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 14:43:34.170020 5122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fc00515d0a41a9a2d7e0c16d810921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } }
I20260926 14:43:34.170351 5122 leader_election.cc:302] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75fc00515d0a41a9a2d7e0c16d810921; no voters:
I20260926 14:43:34.170706 5122 leader_election.cc:288] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:34.170789 5132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:34.171000 5132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [term 1 LEADER]: Becoming Leader. State: Replica: 75fc00515d0a41a9a2d7e0c16d810921, State: Running, Role: LEADER
I20260926 14:43:34.171301 5132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fc00515d0a41a9a2d7e0c16d810921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } }
I20260926 14:43:34.171942 5122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:34.172402 5132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75fc00515d0a41a9a2d7e0c16d810921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fc00515d0a41a9a2d7e0c16d810921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } } }
I20260926 14:43:34.172521 5132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:34.172736 5132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75fc00515d0a41a9a2d7e0c16d810921. Latest consensus state: current_term: 1 leader_uuid: "75fc00515d0a41a9a2d7e0c16d810921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fc00515d0a41a9a2d7e0c16d810921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } } }
I20260926 14:43:34.172809 5132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:34.172880 5136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:34.175868 5136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:34.181520 5136 catalog_manager.cc:1347] Generated new cluster ID: 3f70073d7e7341c394c974d24a0a061c
I20260926 14:43:34.181579 5136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:34.192533 5136 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:34.193993 5136 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:34.207634 5136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75fc00515d0a41a9a2d7e0c16d810921: Generated new TSK 0
I20260926 14:43:34.208424 5136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:34.223837 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 5054
2026-09-26T14:43:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:34 dist-test-slave-x4sn krb5kdc[5153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:34 dist-test-slave-x4sn krb5kdc[5153](info): set up 2 sockets
Sep 26 14:43:34 dist-test-slave-x4sn krb5kdc[5153](info): commencing operation
krb5kdc: starting...
W20260926 14:43:35.177258 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:35 dist-test-slave-x4sn krb5kdc[5153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:43:35.288359 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:35.395128 5166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:35.395375 5166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:35.395418 5166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:35.399295 5166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:35.399348 5166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:35.399365 5166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:35.399395 5166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:35.399422 5166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:35.404552 5166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:35.405543 5166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:35.406490 5166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:35.414206 5171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:35.414276 5172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:35.414216 5174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:35.414387 5166 server_base.cc:1034] running on GCE node
I20260926 14:43:35.414999 5166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:35.416081 5166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:35.417228 5166 hybrid_clock.cc:648] HybridClock initialized: now 1790433815417216 us; error 26 us; skew 500 ppm
Sep 26 14:43:35 dist-test-slave-x4sn krb5kdc[5153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:35.423723 5166 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:43:35.424572 5166 webserver.cc:460] Webserver started at http://127.0.0.1:42033/ using document root <none> and password file <none>
I20260926 14:43:35.425077 5166 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:35.425141 5166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:35.425347 5166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:35.427261 5166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "f2815b35f1a04d08b5665ef3feb8cced"
format_stamp: "Formatted at 2026-09-26 14:43:35 on dist-test-slave-x4sn"
server_key: "15ebea45c6aa11d24bfcfe693c34dff6"
server_key_iv: "d43e4194bd16490d77140371c9e72012"
server_key_version: "encryptionkey@0"
I20260926 14:43:35.427800 5166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "f2815b35f1a04d08b5665ef3feb8cced"
format_stamp: "Formatted at 2026-09-26 14:43:35 on dist-test-slave-x4sn"
server_key: "15ebea45c6aa11d24bfcfe693c34dff6"
server_key_iv: "d43e4194bd16490d77140371c9e72012"
server_key_version: "encryptionkey@0"
I20260926 14:43:35.431273 5166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:35.433544 5181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:35.434655 5166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:35.434763 5166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "f2815b35f1a04d08b5665ef3feb8cced"
format_stamp: "Formatted at 2026-09-26 14:43:35 on dist-test-slave-x4sn"
server_key: "15ebea45c6aa11d24bfcfe693c34dff6"
server_key_iv: "d43e4194bd16490d77140371c9e72012"
server_key_version: "encryptionkey@0"
I20260926 14:43:35.434849 5166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:35.439074 5166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:35.441604 5166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:35.441769 5166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:35.448101 5166 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33505
I20260926 14:43:35.448141 5233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33505 every 8 connection(s)
I20260926 14:43:35.448925 5166 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:35.451401 5234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:35.453589 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 5166
I20260926 14:43:35.453712 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:35.453959 16747 external_mini_cluster.cc:1428] Setting key 3fc1c06fec803bf861d6d443161ef5dc
I20260926 14:43:35.457216 5234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced: Bootstrap starting.
I20260926 14:43:35.458997 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:35.455173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38729 (local address 127.0.0.1:33505)
0926 14:43:35.455557 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:35.455574 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:35.455615 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:43:35.456449 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:35.456471 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:35.456807 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:35.457155 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:35.457722 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:35.457726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:35.457740 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:35.457750 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:35.457802 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:35.457826 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:35.457949 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:35.458009 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 14:43:35.458155 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
I20260926 14:43:35.459653 5234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:35.460489 5234 log.cc:824] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:35.461621 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:35.462337 5234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced: No bootstrap required, opened a new log
I20260926 14:43:35.464797 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:35.463432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38481 (local address 127.0.0.1:33505)
0926 14:43:35.463528 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:35.463532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:35.463689 (+ 157us) server_negotiation.cc:407] Connection header received
0926 14:43:35.463810 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:35.463812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:35.463836 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:35.463894 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:35.464484 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:35.464487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:35.464488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:35.464497 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:35.464519 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:35.464528 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:35.464586 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:35.464676 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 14:43:35.464703 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:43:35.466198 5234 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2815b35f1a04d08b5665ef3feb8cced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } }
I20260926 14:43:35.466398 5234 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:35.466451 5234 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2815b35f1a04d08b5665ef3feb8cced, State: Initialized, Role: FOLLOWER
I20260926 14:43:35.466869 5234 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2815b35f1a04d08b5665ef3feb8cced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } }
I20260926 14:43:35.467022 5234 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:35.467087 5234 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:35.467181 5234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:35.468057 5234 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2815b35f1a04d08b5665ef3feb8cced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } }
I20260926 14:43:35.468333 5234 leader_election.cc:302] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2815b35f1a04d08b5665ef3feb8cced; no voters:
I20260926 14:43:35.468551 5234 leader_election.cc:288] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:35.468647 5244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:35.468833 5244 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [term 1 LEADER]: Becoming Leader. State: Replica: f2815b35f1a04d08b5665ef3feb8cced, State: Running, Role: LEADER
I20260926 14:43:35.469102 5244 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2815b35f1a04d08b5665ef3feb8cced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } }
I20260926 14:43:35.469651 5234 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:35.470572 5245 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2815b35f1a04d08b5665ef3feb8cced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2815b35f1a04d08b5665ef3feb8cced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } } }
I20260926 14:43:35.470695 5245 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:35.470942 5246 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2815b35f1a04d08b5665ef3feb8cced. Latest consensus state: current_term: 1 leader_uuid: "f2815b35f1a04d08b5665ef3feb8cced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2815b35f1a04d08b5665ef3feb8cced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } } }
I20260926 14:43:35.471027 5246 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:35.471120 5253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:35.474062 5253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:35.479368 5253 catalog_manager.cc:1347] Generated new cluster ID: 3a497d5d34614771b503bc01d84701c3
I20260926 14:43:35.479436 5253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:35.494014 5253 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:35.494302 5253 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:35.505267 5253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2815b35f1a04d08b5665ef3feb8cced: Generated new TSK 0
I20260926 14:43:35.505950 5253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:35.515054 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 5166
2026-09-26T14:43:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:35 dist-test-slave-x4sn krb5kdc[5265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:35 dist-test-slave-x4sn krb5kdc[5265](info): set up 2 sockets
Sep 26 14:43:35 dist-test-slave-x4sn krb5kdc[5265](info): commencing operation
krb5kdc: starting...
W20260926 14:43:36.457278 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:36 dist-test-slave-x4sn krb5kdc[5265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:43:36.572093 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:36.681035 5278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:36.681294 5278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:36.681351 5278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:36.685324 5278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:36.685382 5278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:36.685407 5278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:36.685438 5278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:36.685467 5278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:36.690584 5278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:36.691577 5278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:36.692615 5278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:36.700263 5286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:36.700320 5283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:36.700352 5284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:36.700482 5278 server_base.cc:1034] running on GCE node
I20260926 14:43:36.701022 5278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:36.702059 5278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:36.703213 5278 hybrid_clock.cc:648] HybridClock initialized: now 1790433816703202 us; error 28 us; skew 500 ppm
Sep 26 14:43:36 dist-test-slave-x4sn krb5kdc[5265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:36.705726 5278 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:43:36.706497 5278 webserver.cc:460] Webserver started at http://127.0.0.1:41913/ using document root <none> and password file <none>
I20260926 14:43:36.706983 5278 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:36.707041 5278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:36.707257 5278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:36.709237 5278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "50bdb6538171423cb27404d66abdbe66"
format_stamp: "Formatted at 2026-09-26 14:43:36 on dist-test-slave-x4sn"
server_key: "94121687df74043e8be53d1ae8fbb57f"
server_key_iv: "d504b208e375d1055d52092d0016dfd0"
server_key_version: "encryptionkey@0"
I20260926 14:43:36.709757 5278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "50bdb6538171423cb27404d66abdbe66"
format_stamp: "Formatted at 2026-09-26 14:43:36 on dist-test-slave-x4sn"
server_key: "94121687df74043e8be53d1ae8fbb57f"
server_key_iv: "d504b208e375d1055d52092d0016dfd0"
server_key_version: "encryptionkey@0"
I20260926 14:43:36.713217 5278 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260926 14:43:36.715380 5293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:36.716548 5278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 14:43:36.716656 5278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "50bdb6538171423cb27404d66abdbe66"
format_stamp: "Formatted at 2026-09-26 14:43:36 on dist-test-slave-x4sn"
server_key: "94121687df74043e8be53d1ae8fbb57f"
server_key_iv: "d504b208e375d1055d52092d0016dfd0"
server_key_version: "encryptionkey@0"
I20260926 14:43:36.716743 5278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:36.726922 5278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:36.729522 5278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:36.729687 5278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:36.736284 5278 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40251
I20260926 14:43:36.736301 5345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40251 every 8 connection(s)
I20260926 14:43:36.737175 5278 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:36.737448 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 5278
I20260926 14:43:36.737555 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:36.737812 16747 external_mini_cluster.cc:1428] Setting key be383cadf55e2e14a1cf1730c2d19f55
I20260926 14:43:36.740368 5346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 14:43:36 dist-test-slave-x4sn krb5kdc[5265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:43:36.746611 5346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66: Bootstrap starting.
I20260926 14:43:36.749130 5346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:36.749939 5346 log.cc:824] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:36.751789 5346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66: No bootstrap required, opened a new log
I20260926 14:43:36.751863 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:36.739098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:54489 (local address 127.0.0.1:40251)
0926 14:43:36.739527 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:36.739546 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:36.739587 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:43:36.740434 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:36.740457 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:36.740881 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:36.741249 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:36.741883 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:36.743340 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:36.744180 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:36.744837 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:36.745001 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:36.747268 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:36.747273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:36.747286 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:36.747311 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:36.749052 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:36.749535 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:36.749540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:36.749546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:36.749680 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:36.749938 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:36.749940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:36.749942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:36.750406 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:36.750607 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:36.750924 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 14:43:36.751123 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":116,"threads_started":1}
I20260926 14:43:36.754696 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:36.755508 5346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50bdb6538171423cb27404d66abdbe66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } }
I20260926 14:43:36.755739 5346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:36.755802 5346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bdb6538171423cb27404d66abdbe66, State: Initialized, Role: FOLLOWER
I20260926 14:43:36.756186 5346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50bdb6538171423cb27404d66abdbe66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } }
I20260926 14:43:36.756304 5346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:36.756366 5346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:36.756456 5346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:36.757437 5346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50bdb6538171423cb27404d66abdbe66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } }
I20260926 14:43:36.757771 5346 leader_election.cc:302] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50bdb6538171423cb27404d66abdbe66; no voters:
I20260926 14:43:36.758073 5346 leader_election.cc:288] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:36.758164 5356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:36.758371 5356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [term 1 LEADER]: Becoming Leader. State: Replica: 50bdb6538171423cb27404d66abdbe66, State: Running, Role: LEADER
I20260926 14:43:36.758682 5356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50bdb6538171423cb27404d66abdbe66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } }
I20260926 14:43:36.759287 5346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:36.760200 5357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50bdb6538171423cb27404d66abdbe66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50bdb6538171423cb27404d66abdbe66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } } }
I20260926 14:43:36.760327 5357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:36.761242 5358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50bdb6538171423cb27404d66abdbe66. Latest consensus state: current_term: 1 leader_uuid: "50bdb6538171423cb27404d66abdbe66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50bdb6538171423cb27404d66abdbe66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } } }
I20260926 14:43:36.761350 5358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:36.761598 5365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:36.764566 5365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:36.765379 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:36.756570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:52109 (local address 127.0.0.1:40251)
0926 14:43:36.756671 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:36.756675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:36.756903 (+ 228us) server_negotiation.cc:407] Connection header received
0926 14:43:36.756971 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:36.756974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:36.757018 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:36.757094 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:36.757673 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:36.758394 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:36.759440 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:36.759786 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:36.759889 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:36.761615 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:36.761619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:36.761622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:36.761656 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:36.763047 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:36.763584 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:36.763587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:36.763589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:36.763693 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:36.764062 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:36.764066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:36.764067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:36.764257 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:36.764388 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:36.765152 (+ 764us) server_negotiation.cc:299] Negotiation successful
0926 14:43:36.765242 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 14:43:36.769573 5365 catalog_manager.cc:1347] Generated new cluster ID: 29e61ee0aff3491c955a4c7cec2ce6a9
I20260926 14:43:36.769622 5365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:36.789572 5365 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:36.790483 5365 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:36.795727 5365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50bdb6538171423cb27404d66abdbe66: Generated new TSK 0
I20260926 14:43:36.796522 5365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:36.808481 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 5278
W20260926 14:43:36.813372 5263 connection.cc:535] client connection to 127.0.0.1:40251 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40251): Connection reset by peer (error 104)
2026-09-26T14:43:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:36 dist-test-slave-x4sn krb5kdc[5377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:36 dist-test-slave-x4sn krb5kdc[5377](info): set up 2 sockets
Sep 26 14:43:36 dist-test-slave-x4sn krb5kdc[5377](info): commencing operation
krb5kdc: starting...
W20260926 14:43:37.753264 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:37 dist-test-slave-x4sn krb5kdc[5377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:43:37.868330 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:37.974987 5390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:37.975268 5390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:37.975311 5390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:37.979259 5390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:37.979328 5390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:37.979349 5390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:37.979379 5390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:37.979405 5390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:37.984694 5390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:37.985718 5390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:37.986703 5390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:37.994712 5396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:37.994715 5395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:37.994715 5398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:37.994822 5390 server_base.cc:1034] running on GCE node
I20260926 14:43:37.995481 5390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:37.996551 5390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:37.997706 5390 hybrid_clock.cc:648] HybridClock initialized: now 1790433817997689 us; error 30 us; skew 500 ppm
Sep 26 14:43:37 dist-test-slave-x4sn krb5kdc[5377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:38.000166 5390 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:43:38.000945 5390 webserver.cc:460] Webserver started at http://127.0.0.1:39993/ using document root <none> and password file <none>
I20260926 14:43:38.001433 5390 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:38.001487 5390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:38.001641 5390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:38.003553 5390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "af7d7c169e3845bdbe135b8b5e4d7aab"
format_stamp: "Formatted at 2026-09-26 14:43:37 on dist-test-slave-x4sn"
server_key: "2a33c007af2ef0f11d80cdfae79953ac"
server_key_iv: "a5b6f82e5b9e81af75f485910a405fd4"
server_key_version: "encryptionkey@0"
I20260926 14:43:38.004094 5390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "af7d7c169e3845bdbe135b8b5e4d7aab"
format_stamp: "Formatted at 2026-09-26 14:43:37 on dist-test-slave-x4sn"
server_key: "2a33c007af2ef0f11d80cdfae79953ac"
server_key_iv: "a5b6f82e5b9e81af75f485910a405fd4"
server_key_version: "encryptionkey@0"
I20260926 14:43:38.007555 5390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:38.009883 5405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:38.011116 5390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:38.011221 5390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "af7d7c169e3845bdbe135b8b5e4d7aab"
format_stamp: "Formatted at 2026-09-26 14:43:37 on dist-test-slave-x4sn"
server_key: "2a33c007af2ef0f11d80cdfae79953ac"
server_key_iv: "a5b6f82e5b9e81af75f485910a405fd4"
server_key_version: "encryptionkey@0"
I20260926 14:43:38.011302 5390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:38.025255 5390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:38.027828 5390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:38.027979 5390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:38.034467 5390 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36429
I20260926 14:43:38.034495 5457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36429 every 8 connection(s)
I20260926 14:43:38.035346 5390 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:38.037973 5458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:38.043166 5458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab: Bootstrap starting.
I20260926 14:43:38.044170 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 5390
I20260926 14:43:38.044315 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:38.044618 16747 external_mini_cluster.cc:1428] Setting key 0019ea2d8504dadb37aae7d0cdb37986
I20260926 14:43:38.045855 5458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:38.046583 5458 log.cc:824] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:38.048426 5458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab: No bootstrap required, opened a new log
I20260926 14:43:38.051640 5458 raft_consensus.cc:348] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af7d7c169e3845bdbe135b8b5e4d7aab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36429 } }
I20260926 14:43:38.051894 5458 raft_consensus.cc:374] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:38.051949 5458 raft_consensus.cc:729] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7d7c169e3845bdbe135b8b5e4d7aab, State: Initialized, Role: FOLLOWER
I20260926 14:43:38.052415 5458 consensus_queue.cc:260] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af7d7c169e3845bdbe135b8b5e4d7aab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36429 } }
I20260926 14:43:38.052539 5458 raft_consensus.cc:388] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:38.052597 5458 raft_consensus.cc:482] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:38.052690 5458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:38.052999 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:38.045684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:36461 (local address 127.0.0.1:36429)
0926 14:43:38.046110 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:38.046126 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:38.046171 (+ 45us) server_negotiation.cc:407] Connection header received
0926 14:43:38.046819 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:38.046837 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:38.047256 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:38.047577 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:38.048007 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:38.048981 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:38.049854 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:38.050321 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:38.050463 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:38.051263 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:38.051268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:38.051280 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:38.051640 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:38.051733 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:38.051759 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:38.051871 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:38.052050 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 14:43:38.052201 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":119,"threads_started":1}
I20260926 14:43:38.053582 5458 raft_consensus.cc:504] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af7d7c169e3845bdbe135b8b5e4d7aab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36429 } }
I20260926 14:43:38.053886 5458 leader_election.cc:302] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af7d7c169e3845bdbe135b8b5e4d7aab; no voters:
I20260926 14:43:38.054162 5458 leader_election.cc:288] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:38.054250 5463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:38.054404 5463 raft_consensus.cc:686] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [term 1 LEADER]: Becoming Leader. State: Replica: af7d7c169e3845bdbe135b8b5e4d7aab, State: Running, Role: LEADER
I20260926 14:43:38.054754 5463 consensus_queue.cc:237] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af7d7c169e3845bdbe135b8b5e4d7aab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36429 } }
I20260926 14:43:38.055374 5458 sys_catalog.cc:564] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:38.055613 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:38.056346 5464 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af7d7c169e3845bdbe135b8b5e4d7aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af7d7c169e3845bdbe135b8b5e4d7aab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36429 } } }
I20260926 14:43:38.056519 5464 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:38.056818 5464 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [sys.catalog]: SysCatalogTable state changed. Reason: New leader af7d7c169e3845bdbe135b8b5e4d7aab. Latest consensus state: current_term: 1 leader_uuid: "af7d7c169e3845bdbe135b8b5e4d7aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af7d7c169e3845bdbe135b8b5e4d7aab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36429 } } }
I20260926 14:43:38.056957 5464 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:38.057945 5472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:38.060971 5472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:38.061630 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:38.057736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:45553 (local address 127.0.0.1:36429)
0926 14:43:38.057819 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:38.057823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:38.058219 (+ 396us) server_negotiation.cc:407] Connection header received
0926 14:43:38.058328 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:38.058330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:38.058367 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:38.058426 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:38.058698 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:38.059188 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:38.060093 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:38.060339 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:38.060423 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:38.061216 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:38.061219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:38.061222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:38.061234 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:38.061258 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:38.061269 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:38.061335 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:38.061478 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 14:43:38.061510 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:43:38.066191 5472 catalog_manager.cc:1347] Generated new cluster ID: 6a1e577a980d4c0a8a0a3639e31556b3
I20260926 14:43:38.066241 5472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:38.077312 5472 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:38.078123 5472 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:38.082012 5472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af7d7c169e3845bdbe135b8b5e4d7aab: Generated new TSK 0
I20260926 14:43:38.082574 5472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:38.087160 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 5390
W20260926 14:43:38.091737 5375 connection.cc:535] client connection to 127.0.0.1:36429 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36429): Connection reset by peer (error 104)
2026-09-26T14:43:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:38 dist-test-slave-x4sn krb5kdc[5489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:38 dist-test-slave-x4sn krb5kdc[5489](info): set up 2 sockets
Sep 26 14:43:38 dist-test-slave-x4sn krb5kdc[5489](info): commencing operation
krb5kdc: starting...
W20260926 14:43:39.029654 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:39 dist-test-slave-x4sn krb5kdc[5489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:43:39.140709 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44433
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:39.247445 5502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:39.247752 5502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:39.247862 5502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:39.251796 5502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:39.251854 5502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:39.251873 5502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:39.251888 5502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:39.251915 5502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:39.257040 5502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:39.258064 5502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:39.259030 5502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:39.266958 5507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:39.266995 5510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:39.267001 5502 server_base.cc:1034] running on GCE node
W20260926 14:43:39.266958 5508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:39.267753 5502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:39.268759 5502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:39.269903 5502 hybrid_clock.cc:648] HybridClock initialized: now 1790433819269881 us; error 33 us; skew 500 ppm
Sep 26 14:43:39 dist-test-slave-x4sn krb5kdc[5489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:39.272390 5502 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:43:39.273205 5502 webserver.cc:460] Webserver started at http://127.0.0.1:33247/ using document root <none> and password file <none>
I20260926 14:43:39.273705 5502 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:39.273770 5502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:39.273983 5502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:39.275949 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "ee98925e61204a3dbf04271e2421c5c9"
format_stamp: "Formatted at 2026-09-26 14:43:39 on dist-test-slave-x4sn"
server_key: "c302ee1801451d5d0cfa41b2d1b69a7b"
server_key_iv: "211ef71a0197134390945d41c9f26762"
server_key_version: "encryptionkey@0"
I20260926 14:43:39.276454 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "ee98925e61204a3dbf04271e2421c5c9"
format_stamp: "Formatted at 2026-09-26 14:43:39 on dist-test-slave-x4sn"
server_key: "c302ee1801451d5d0cfa41b2d1b69a7b"
server_key_iv: "211ef71a0197134390945d41c9f26762"
server_key_version: "encryptionkey@0"
I20260926 14:43:39.279870 5502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:43:39.282080 5517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:39.283167 5502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:39.283272 5502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "ee98925e61204a3dbf04271e2421c5c9"
format_stamp: "Formatted at 2026-09-26 14:43:39 on dist-test-slave-x4sn"
server_key: "c302ee1801451d5d0cfa41b2d1b69a7b"
server_key_iv: "211ef71a0197134390945d41c9f26762"
server_key_version: "encryptionkey@0"
I20260926 14:43:39.283360 5502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:39.287544 5502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:39.290026 5502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:39.290182 5502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:39.296633 5502 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44433
I20260926 14:43:39.296638 5569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44433 every 8 connection(s)
I20260926 14:43:39.297458 5502 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:39.299962 5570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:39.305047 5570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9: Bootstrap starting.
I20260926 14:43:39.305693 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 5502
I20260926 14:43:39.305823 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:39.306107 16747 external_mini_cluster.cc:1428] Setting key e928c4322b6f377726d06b98fb9cb051
I20260926 14:43:39.307804 5570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:39.308501 5570 log.cc:824] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:39.310343 5570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9: No bootstrap required, opened a new log
I20260926 14:43:39.310931 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:39.307314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:52677 (local address 127.0.0.1:44433)
0926 14:43:39.307762 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:39.307775 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:39.307810 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:43:39.308435 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:39.308451 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:39.308759 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:39.309055 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:39.309774 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:39.309779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:39.309790 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:39.309800 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:39.309853 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:39.309871 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:39.309963 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:39.310081 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 14:43:39.310238 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":102,"threads_started":1}
I20260926 14:43:39.313441 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:39.313951 5570 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee98925e61204a3dbf04271e2421c5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44433 } }
I20260926 14:43:39.314148 5570 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:39.314210 5570 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee98925e61204a3dbf04271e2421c5c9, State: Initialized, Role: FOLLOWER
I20260926 14:43:39.314597 5570 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee98925e61204a3dbf04271e2421c5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44433 } }
I20260926 14:43:39.314745 5570 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:39.314811 5570 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:39.314906 5570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:39.315922 5570 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee98925e61204a3dbf04271e2421c5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44433 } }
I20260926 14:43:39.316272 5570 leader_election.cc:302] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee98925e61204a3dbf04271e2421c5c9; no voters:
I20260926 14:43:39.316581 5570 leader_election.cc:288] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:39.316668 5580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:39.316893 5580 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [term 1 LEADER]: Becoming Leader. State: Replica: ee98925e61204a3dbf04271e2421c5c9, State: Running, Role: LEADER
I20260926 14:43:39.316984 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:39.315393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:54953 (local address 127.0.0.1:44433)
0926 14:43:39.315492 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:39.315496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:39.315702 (+ 206us) server_negotiation.cc:407] Connection header received
0926 14:43:39.315804 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:39.315806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:39.315836 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:39.315896 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:39.316600 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:39.316603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:39.316606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:39.316617 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:39.316643 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:39.316655 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:39.316724 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:39.316802 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 14:43:39.316838 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:43:39.317235 5580 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee98925e61204a3dbf04271e2421c5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44433 } }
I20260926 14:43:39.317806 5570 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:39.318442 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee98925e61204a3dbf04271e2421c5c9. Latest consensus state: current_term: 1 leader_uuid: "ee98925e61204a3dbf04271e2421c5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee98925e61204a3dbf04271e2421c5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44433 } } }
I20260926 14:43:39.318619 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:39.318812 5581 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee98925e61204a3dbf04271e2421c5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee98925e61204a3dbf04271e2421c5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44433 } } }
I20260926 14:43:39.318910 5581 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:39.319015 5589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:39.322045 5589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:39.326990 5589 catalog_manager.cc:1347] Generated new cluster ID: f0f0dcb6675a49f598ac8637fbfa9875
I20260926 14:43:39.327045 5589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:39.346829 5589 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:39.347127 5589 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:39.351030 5589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee98925e61204a3dbf04271e2421c5c9: Generated new TSK 0
I20260926 14:43:39.351629 5589 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:39.360344 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 5502
2026-09-26T14:43:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:39 dist-test-slave-x4sn krb5kdc[5601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:39 dist-test-slave-x4sn krb5kdc[5601](info): set up 2 sockets
Sep 26 14:43:39 dist-test-slave-x4sn krb5kdc[5601](info): commencing operation
krb5kdc: starting...
W20260926 14:43:40.309226 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:40 dist-test-slave-x4sn krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:43:40.430199 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:40.536675 5614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:40.536938 5614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:40.536980 5614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:40.541215 5614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:40.541283 5614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:40.541303 5614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:40.541319 5614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:40.541343 5614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:40.546844 5614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:40.547922 5614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:40.548902 5614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:40.557103 5620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:40.557060 5622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:40.557158 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:40.557384 5614 server_base.cc:1034] running on GCE node
I20260926 14:43:40.557978 5614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:40.559051 5614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:40.560202 5614 hybrid_clock.cc:648] HybridClock initialized: now 1790433820560179 us; error 39 us; skew 500 ppm
Sep 26 14:43:40 dist-test-slave-x4sn krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:40.571362 5614 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:43:40.572348 5614 webserver.cc:460] Webserver started at http://127.0.0.1:34819/ using document root <none> and password file <none>
I20260926 14:43:40.572898 5614 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:40.572966 5614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:40.573179 5614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:40.575258 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "31754964b8e841b0beb5f15f9771b72f"
format_stamp: "Formatted at 2026-09-26 14:43:40 on dist-test-slave-x4sn"
server_key: "3b70cc110546d8e78032b27c2fa4eb29"
server_key_iv: "698a56604d4aad2dcba428326d5f1d9a"
server_key_version: "encryptionkey@0"
I20260926 14:43:40.575827 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "31754964b8e841b0beb5f15f9771b72f"
format_stamp: "Formatted at 2026-09-26 14:43:40 on dist-test-slave-x4sn"
server_key: "3b70cc110546d8e78032b27c2fa4eb29"
server_key_iv: "698a56604d4aad2dcba428326d5f1d9a"
server_key_version: "encryptionkey@0"
I20260926 14:43:40.579769 5614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:43:40.582297 5629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:40.583519 5614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:40.583633 5614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "31754964b8e841b0beb5f15f9771b72f"
format_stamp: "Formatted at 2026-09-26 14:43:40 on dist-test-slave-x4sn"
server_key: "3b70cc110546d8e78032b27c2fa4eb29"
server_key_iv: "698a56604d4aad2dcba428326d5f1d9a"
server_key_version: "encryptionkey@0"
I20260926 14:43:40.583752 5614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:40.599475 5614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:40.602392 5614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:40.602579 5614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:40.609632 5614 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45675
I20260926 14:43:40.609648 5681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45675 every 8 connection(s)
I20260926 14:43:40.610575 5614 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:40.613276 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:40.615820 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 5614
I20260926 14:43:40.615968 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:40.616258 16747 external_mini_cluster.cc:1428] Setting key 115ae63b2f6cf2cdaa189856058ec103
I20260926 14:43:40.619254 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f: Bootstrap starting.
I20260926 14:43:40.621753 5682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:40.622587 5682 log.cc:824] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:40.624812 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f: No bootstrap required, opened a new log
Sep 26 14:43:40 dist-test-slave-x4sn krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:43:40.627694 5682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31754964b8e841b0beb5f15f9771b72f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260926 14:43:40.627928 5682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:40.627990 5682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31754964b8e841b0beb5f15f9771b72f, State: Initialized, Role: FOLLOWER
I20260926 14:43:40.628439 5682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31754964b8e841b0beb5f15f9771b72f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260926 14:43:40.628599 5682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:40.628671 5682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:40.628764 5682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:40.629809 5682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31754964b8e841b0beb5f15f9771b72f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260926 14:43:40.630119 5682 leader_election.cc:302] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31754964b8e841b0beb5f15f9771b72f; no voters:
I20260926 14:43:40.630393 5682 leader_election.cc:288] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:40.630498 5687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:40.630605 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:40.617640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:59277 (local address 127.0.0.1:45675)
0926 14:43:40.618026 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:40.618041 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:40.618076 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:43:40.618702 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:40.618719 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:40.619055 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:40.619345 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:40.620196 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:40.621181 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:40.622309 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:40.622800 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:40.622936 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:40.625532 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:40.625537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:40.625549 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:40.625577 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:40.627516 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:40.628071 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:40.628086 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:40.628093 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:40.628215 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:40.628558 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:40.628561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:40.628563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:40.629021 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:40.629205 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:40.629486 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 14:43:40.629706 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
I20260926 14:43:40.630726 5687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [term 1 LEADER]: Becoming Leader. State: Replica: 31754964b8e841b0beb5f15f9771b72f, State: Running, Role: LEADER
I20260926 14:43:40.631026 5687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31754964b8e841b0beb5f15f9771b72f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260926 14:43:40.631562 5682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:40.632292 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31754964b8e841b0beb5f15f9771b72f. Latest consensus state: current_term: 1 leader_uuid: "31754964b8e841b0beb5f15f9771b72f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31754964b8e841b0beb5f15f9771b72f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } } }
I20260926 14:43:40.632407 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:40.632701 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31754964b8e841b0beb5f15f9771b72f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31754964b8e841b0beb5f15f9771b72f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } } }
I20260926 14:43:40.632779 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:40.633074 5697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:40.634008 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:40.636749 5697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:40.642608 5697 catalog_manager.cc:1347] Generated new cluster ID: 07de552606b7409c8341dfddd06b0660
I20260926 14:43:40.642669 5697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:40.644256 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:40.636094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:42049 (local address 127.0.0.1:45675)
0926 14:43:40.636201 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:40.636205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:40.636427 (+ 222us) server_negotiation.cc:407] Connection header received
0926 14:43:40.636549 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:40.636552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:40.636599 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:40.636677 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:40.637431 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:40.638220 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:40.639182 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:40.639419 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:40.639503 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:40.640851 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:40.640855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:40.640858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:40.640885 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:40.642424 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:40.642938 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:40.642942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:40.642943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:40.643027 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:40.643388 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:40.643392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:40.643394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:40.643594 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:40.643783 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:40.643993 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 14:43:40.644090 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":24}
I20260926 14:43:40.649964 5697 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:40.650808 5697 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:40.660472 5697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31754964b8e841b0beb5f15f9771b72f: Generated new TSK 0
I20260926 14:43:40.661156 5697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:40.664642 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 5614
W20260926 14:43:40.669163 5599 connection.cc:535] client connection to 127.0.0.1:45675 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45675): Connection reset by peer (error 104)
2026-09-26T14:43:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:40 dist-test-slave-x4sn krb5kdc[5714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:40 dist-test-slave-x4sn krb5kdc[5714](info): set up 2 sockets
Sep 26 14:43:40 dist-test-slave-x4sn krb5kdc[5714](info): commencing operation
krb5kdc: starting...
W20260926 14:43:41.613212 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:41 dist-test-slave-x4sn krb5kdc[5714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:43:41.728585 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:41.834164 5727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:41.834437 5727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:41.834496 5727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:41.838483 5727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:41.838551 5727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:41.838578 5727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:41.838599 5727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:41.838618 5727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:41.843931 5727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:41.844977 5727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:41.845964 5727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:41.853945 5735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:41.853956 5732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:41.854142 5727 server_base.cc:1034] running on GCE node
W20260926 14:43:41.853960 5733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:41.854709 5727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:41.855834 5727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:41.856988 5727 hybrid_clock.cc:648] HybridClock initialized: now 1790433821856973 us; error 26 us; skew 500 ppm
Sep 26 14:43:41 dist-test-slave-x4sn krb5kdc[5714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:41.859540 5727 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:43:41.860419 5727 webserver.cc:460] Webserver started at http://127.0.0.1:42275/ using document root <none> and password file <none>
I20260926 14:43:41.860962 5727 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:41.861028 5727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:41.861236 5727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:41.863183 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "12c960c2e75f4223a750723eb2d495ff"
format_stamp: "Formatted at 2026-09-26 14:43:41 on dist-test-slave-x4sn"
server_key: "a1f51f7eb7c750b65e3b6a645121b42b"
server_key_iv: "e3618c217063d60d00ed889fd8dd9b8e"
server_key_version: "encryptionkey@0"
I20260926 14:43:41.863727 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "12c960c2e75f4223a750723eb2d495ff"
format_stamp: "Formatted at 2026-09-26 14:43:41 on dist-test-slave-x4sn"
server_key: "a1f51f7eb7c750b65e3b6a645121b42b"
server_key_iv: "e3618c217063d60d00ed889fd8dd9b8e"
server_key_version: "encryptionkey@0"
I20260926 14:43:41.867138 5727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 14:43:41.869364 5742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:41.870587 5727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:43:41.870704 5727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "12c960c2e75f4223a750723eb2d495ff"
format_stamp: "Formatted at 2026-09-26 14:43:41 on dist-test-slave-x4sn"
server_key: "a1f51f7eb7c750b65e3b6a645121b42b"
server_key_iv: "e3618c217063d60d00ed889fd8dd9b8e"
server_key_version: "encryptionkey@0"
I20260926 14:43:41.870798 5727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:41.885435 5727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:41.888429 5727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:41.888612 5727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:41.895198 5727 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34815
I20260926 14:43:41.895212 5794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34815 every 8 connection(s)
I20260926 14:43:41.896137 5727 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:41.898631 5795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:41.903834 5795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff: Bootstrap starting.
I20260926 14:43:41.904161 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 5727
I20260926 14:43:41.904292 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:41.904557 16747 external_mini_cluster.cc:1428] Setting key 8bdf35549ded7a9c7411404e7b0b9e01
I20260926 14:43:41.906263 5795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:41.906947 5795 log.cc:824] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:41.908968 5795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff: No bootstrap required, opened a new log
I20260926 14:43:41.913019 5795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c960c2e75f4223a750723eb2d495ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34815 } }
I20260926 14:43:41.913250 5795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:41.913306 5795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c960c2e75f4223a750723eb2d495ff, State: Initialized, Role: FOLLOWER
I20260926 14:43:41.913702 5795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c960c2e75f4223a750723eb2d495ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34815 } }
I20260926 14:43:41.913827 5795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:41.913890 5795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:41.913970 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:41.905869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:50631 (local address 127.0.0.1:34815)
0926 14:43:41.906276 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:41.906291 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:41.906328 (+ 37us) server_negotiation.cc:407] Connection header received
0926 14:43:41.906978 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:41.906995 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:41.907463 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:41.907829 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:41.908170 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:41.909159 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:41.910380 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:41.910878 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:41.910994 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:41.911939 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:41.911946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:41.911964 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:41.912500 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:41.912574 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:41.912605 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:41.912732 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:41.912943 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 14:43:41.913088 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":127,"threads_started":1}
I20260926 14:43:41.913983 5795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:41.914970 5795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c960c2e75f4223a750723eb2d495ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34815 } }
I20260926 14:43:41.915335 5795 leader_election.cc:302] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12c960c2e75f4223a750723eb2d495ff; no voters:
I20260926 14:43:41.915724 5795 leader_election.cc:288] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:41.915795 5800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:41.915995 5800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [term 1 LEADER]: Becoming Leader. State: Replica: 12c960c2e75f4223a750723eb2d495ff, State: Running, Role: LEADER
I20260926 14:43:41.916330 5800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c960c2e75f4223a750723eb2d495ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34815 } }
I20260926 14:43:41.917053 5795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:41.917152 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:41.917872 5801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12c960c2e75f4223a750723eb2d495ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c960c2e75f4223a750723eb2d495ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34815 } } }
I20260926 14:43:41.917999 5801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:41.918216 5802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c960c2e75f4223a750723eb2d495ff. Latest consensus state: current_term: 1 leader_uuid: "12c960c2e75f4223a750723eb2d495ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c960c2e75f4223a750723eb2d495ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34815 } } }
I20260926 14:43:41.918308 5802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:41.918565 5809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:41.921527 5809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:41.926712 5809 catalog_manager.cc:1347] Generated new cluster ID: da7ba1037b18459dbe5e41455abc8c86
I20260926 14:43:41.926776 5809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:41.928666 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:41.923657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:48727 (local address 127.0.0.1:34815)
0926 14:43:41.923802 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:41.923806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:41.924158 (+ 352us) server_negotiation.cc:407] Connection header received
0926 14:43:41.924279 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:41.924282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:41.924332 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:41.924424 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:41.924709 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:41.925511 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:41.926757 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:41.927122 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:41.927198 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:41.928247 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:41.928250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:41.928254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:41.928266 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:41.928293 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:41.928305 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:41.928380 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:41.928508 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 14:43:41.928539 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 14:43:41.935194 5809 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:41.936141 5809 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:41.940217 5809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12c960c2e75f4223a750723eb2d495ff: Generated new TSK 0
I20260926 14:43:41.940798 5809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:41.950067 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 5727
W20260926 14:43:41.954672 5712 connection.cc:535] client connection to 127.0.0.1:34815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34815): Connection reset by peer (error 104)
2026-09-26T14:43:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:41 dist-test-slave-x4sn krb5kdc[5826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:41 dist-test-slave-x4sn krb5kdc[5826](info): set up 2 sockets
Sep 26 14:43:41 dist-test-slave-x4sn krb5kdc[5826](info): commencing operation
krb5kdc: starting...
W20260926 14:43:42.897207 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:42 dist-test-slave-x4sn krb5kdc[5826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:43:43.011067 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:43.117712 5839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:43.117990 5839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:43.118049 5839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:43.122274 5839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:43.122345 5839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:43.122371 5839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:43.122394 5839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:43.122423 5839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:43.127815 5839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37417
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:43.128888 5839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:43.129908 5839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:43.138057 5844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:43.138065 5845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:43.138123 5847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:43.138139 5839 server_base.cc:1034] running on GCE node
I20260926 14:43:43.138900 5839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:43.140033 5839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:43.141201 5839 hybrid_clock.cc:648] HybridClock initialized: now 1790433823141182 us; error 30 us; skew 500 ppm
Sep 26 14:43:43 dist-test-slave-x4sn krb5kdc[5826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:43.143783 5839 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:43:43.144650 5839 webserver.cc:460] Webserver started at http://127.0.0.1:41247/ using document root <none> and password file <none>
I20260926 14:43:43.145154 5839 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:43.145210 5839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:43.145377 5839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:43.147339 5839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "2b0dafe199cb4b25a88d5b33af601dcd"
format_stamp: "Formatted at 2026-09-26 14:43:43 on dist-test-slave-x4sn"
server_key: "9796d57bd7909d2e1ed550c65b348b2a"
server_key_iv: "3cd9292d4018df2d27674b567b3814cf"
server_key_version: "encryptionkey@0"
I20260926 14:43:43.147877 5839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "2b0dafe199cb4b25a88d5b33af601dcd"
format_stamp: "Formatted at 2026-09-26 14:43:43 on dist-test-slave-x4sn"
server_key: "9796d57bd7909d2e1ed550c65b348b2a"
server_key_iv: "3cd9292d4018df2d27674b567b3814cf"
server_key_version: "encryptionkey@0"
I20260926 14:43:43.151290 5839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:43:43.153658 5854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:43.154867 5839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:43.154976 5839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "2b0dafe199cb4b25a88d5b33af601dcd"
format_stamp: "Formatted at 2026-09-26 14:43:43 on dist-test-slave-x4sn"
server_key: "9796d57bd7909d2e1ed550c65b348b2a"
server_key_iv: "3cd9292d4018df2d27674b567b3814cf"
server_key_version: "encryptionkey@0"
I20260926 14:43:43.155086 5839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:43.159597 5839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:43.162393 5839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:43.162551 5839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:43.169494 5839 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37417
I20260926 14:43:43.169507 5906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37417 every 8 connection(s)
I20260926 14:43:43.170428 5839 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:43.173164 5907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:43.176719 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 5839
I20260926 14:43:43.176841 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:43.177093 16747 external_mini_cluster.cc:1428] Setting key bdbcff51fdbab70434ff7aec711ea100
I20260926 14:43:43.178946 5907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd: Bootstrap starting.
I20260926 14:43:43.181212 5907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:43.182003 5907 log.cc:824] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:43.182189 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:43.178403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:43695 (local address 127.0.0.1:37417)
0926 14:43:43.178843 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:43.178858 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:43.178895 (+ 37us) server_negotiation.cc:407] Connection header received
0926 14:43:43.179566 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:43.179585 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:43.179912 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:43.180227 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:43.180991 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:43.180996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:43.181008 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:43.181018 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:43.181075 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:43.181104 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:43.181194 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:43.181320 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 14:43:43.181474 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":139,"threads_started":1}
I20260926 14:43:43.184644 5907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd: No bootstrap required, opened a new log
I20260926 14:43:43.184751 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:43.188068 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:43.186578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:40945 (local address 127.0.0.1:37417)
0926 14:43:43.186687 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:43.186691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:43.186886 (+ 195us) server_negotiation.cc:407] Connection header received
0926 14:43:43.186984 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:43.186987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:43.187016 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:43.187079 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:43.187728 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:43.187731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:43.187733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:43.187744 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:43.187769 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:43.187779 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:43.187846 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:43.187920 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 14:43:43.187958 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:43:43.188390 5907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0dafe199cb4b25a88d5b33af601dcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37417 } }
I20260926 14:43:43.188623 5907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:43.188678 5907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0dafe199cb4b25a88d5b33af601dcd, State: Initialized, Role: FOLLOWER
I20260926 14:43:43.189121 5907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0dafe199cb4b25a88d5b33af601dcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37417 } }
I20260926 14:43:43.189257 5907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:43.189319 5907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:43.189415 5907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:43.190277 5907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0dafe199cb4b25a88d5b33af601dcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37417 } }
I20260926 14:43:43.190588 5907 leader_election.cc:302] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b0dafe199cb4b25a88d5b33af601dcd; no voters:
I20260926 14:43:43.190865 5907 leader_election.cc:288] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:43.190945 5917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:43.191141 5917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [term 1 LEADER]: Becoming Leader. State: Replica: 2b0dafe199cb4b25a88d5b33af601dcd, State: Running, Role: LEADER
I20260926 14:43:43.191433 5917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0dafe199cb4b25a88d5b33af601dcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37417 } }
I20260926 14:43:43.192090 5907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:43.192978 5919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b0dafe199cb4b25a88d5b33af601dcd. Latest consensus state: current_term: 1 leader_uuid: "2b0dafe199cb4b25a88d5b33af601dcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0dafe199cb4b25a88d5b33af601dcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37417 } } }
I20260926 14:43:43.193135 5919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:43.193415 5918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b0dafe199cb4b25a88d5b33af601dcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0dafe199cb4b25a88d5b33af601dcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37417 } } }
I20260926 14:43:43.193506 5918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:43.193490 5926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:43.196323 5926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:43.201679 5926 catalog_manager.cc:1347] Generated new cluster ID: 4231de3613fc4f61b6b70ce2002133db
I20260926 14:43:43.201741 5926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:43.233179 5926 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:43.233482 5926 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:43.241685 5926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b0dafe199cb4b25a88d5b33af601dcd: Generated new TSK 0
I20260926 14:43:43.242365 5926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:43.254000 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 5839
2026-09-26T14:43:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:43 dist-test-slave-x4sn krb5kdc[5938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:43 dist-test-slave-x4sn krb5kdc[5938](info): set up 2 sockets
Sep 26 14:43:43 dist-test-slave-x4sn krb5kdc[5938](info): commencing operation
krb5kdc: starting...
W20260926 14:43:44.205199 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:44 dist-test-slave-x4sn krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:43:44.316280 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:39301
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:34735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:39301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:44.422675 5952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:44.422947 5952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:44.423004 5952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:44.427026 5952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:44.427095 5952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:44.427120 5952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:44.427141 5952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:44.427167 5952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:44.432327 5952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:34735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:39301
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:39301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:44.433346 5952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:44.434340 5952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:44.442351 5957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:44.442344 5958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:44.442344 5960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:44.442462 5952 server_base.cc:1034] running on GCE node
I20260926 14:43:44.443113 5952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:44.444180 5952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:44.445351 5952 hybrid_clock.cc:648] HybridClock initialized: now 1790433824445324 us; error 37 us; skew 500 ppm
Sep 26 14:43:44 dist-test-slave-x4sn krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433824, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:44.447916 5952 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:43:44.448745 5952 webserver.cc:460] Webserver started at http://127.16.90.254:42791/ using document root <none> and password file <none>
I20260926 14:43:44.449252 5952 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:44.449316 5952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:44.449519 5952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:44.451542 5952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "4f5d120d22054a44b3bd8d4072b91b37"
format_stamp: "Formatted at 2026-09-26 14:43:44 on dist-test-slave-x4sn"
server_key: "ccdc1c2e011e91d4116f657470aa3e6b"
server_key_iv: "c8017627d29abe263e06409bc34e10d3"
server_key_version: "encryptionkey@0"
I20260926 14:43:44.452083 5952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "4f5d120d22054a44b3bd8d4072b91b37"
format_stamp: "Formatted at 2026-09-26 14:43:44 on dist-test-slave-x4sn"
server_key: "ccdc1c2e011e91d4116f657470aa3e6b"
server_key_iv: "c8017627d29abe263e06409bc34e10d3"
server_key_version: "encryptionkey@0"
I20260926 14:43:44.455474 5952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:44.457720 5967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:44.458854 5952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:44.458969 5952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "4f5d120d22054a44b3bd8d4072b91b37"
format_stamp: "Formatted at 2026-09-26 14:43:44 on dist-test-slave-x4sn"
server_key: "ccdc1c2e011e91d4116f657470aa3e6b"
server_key_iv: "c8017627d29abe263e06409bc34e10d3"
server_key_version: "encryptionkey@0"
I20260926 14:43:44.459064 5952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:44.482664 5952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:44.485955 5952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:44.486145 5952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:44.492931 5952 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:39301
I20260926 14:43:44.492950 6019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:39301 every 8 connection(s)
I20260926 14:43:44.493857 5952 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:44.496551 6020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:44.501734 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 5952
I20260926 14:43:44.501816 6020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37: Bootstrap starting.
I20260926 14:43:44.501870 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:44.502111 16747 external_mini_cluster.cc:1428] Setting key e6f636042b34bbfe3b454f5e5a801441
I20260926 14:43:44.504215 6020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:44.504894 6020 log.cc:824] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:44.506752 6020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37: No bootstrap required, opened a new log
Sep 26 14:43:44 dist-test-slave-x4sn krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:44.510154 6020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5d120d22054a44b3bd8d4072b91b37" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39301 } }
I20260926 14:43:44.510406 6020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:44.510470 6020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5d120d22054a44b3bd8d4072b91b37, State: Initialized, Role: FOLLOWER
I20260926 14:43:44.510910 6020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5d120d22054a44b3bd8d4072b91b37" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39301 } }
I20260926 14:43:44.511075 6020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:44.511155 6020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:44.511261 6020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:44.512315 6020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5d120d22054a44b3bd8d4072b91b37" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39301 } }
I20260926 14:43:44.512666 6020 leader_election.cc:302] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f5d120d22054a44b3bd8d4072b91b37; no voters:
I20260926 14:43:44.512989 6020 leader_election.cc:288] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:44.513095 6025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:44.513335 6025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [term 1 LEADER]: Becoming Leader. State: Replica: 4f5d120d22054a44b3bd8d4072b91b37, State: Running, Role: LEADER
I20260926 14:43:44.513710 6025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5d120d22054a44b3bd8d4072b91b37" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39301 } }
I20260926 14:43:44.514237 6020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:44.515288 6026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f5d120d22054a44b3bd8d4072b91b37. Latest consensus state: current_term: 1 leader_uuid: "4f5d120d22054a44b3bd8d4072b91b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5d120d22054a44b3bd8d4072b91b37" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39301 } } }
I20260926 14:43:44.515434 6026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:44.515308 6027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f5d120d22054a44b3bd8d4072b91b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5d120d22054a44b3bd8d4072b91b37" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39301 } } }
I20260926 14:43:44.515807 6027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:44.516000 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:44.503297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45157 (local address 127.16.90.254:39301)
0926 14:43:44.503711 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:44.503728 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:44.503763 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:43:44.504386 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:44.504403 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:44.504740 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:44.505018 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:44.505800 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:44.506773 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:44.507641 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:44.508143 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:44.508295 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:44.510795 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:44.510800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:44.510815 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:44.510844 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:44.512727 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:44.513235 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:44.513241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:44.513246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:44.513378 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:44.513710 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:44.513713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:44.513714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:44.514186 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:44.514388 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:44.514728 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 14:43:44.514940 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260926 14:43:44.518960 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 14:43:44.519099 6040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:43:44.519173 6040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:43:44.519399 6034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:44.520368 6034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:44.526008 6034 catalog_manager.cc:1347] Generated new cluster ID: f4864c361e7a4edcac7bc5cf9254035d
I20260926 14:43:44.526085 6034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:44.531353 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:44.521143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54935 (local address 127.16.90.254:39301)
0926 14:43:44.521241 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:44.521245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:44.521457 (+ 212us) server_negotiation.cc:407] Connection header received
0926 14:43:44.521629 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:44.521632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:44.521676 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:44.521741 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:44.522680 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:44.523839 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:44.524972 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:44.525304 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:44.525406 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:44.528520 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:44.528523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:44.528525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:44.528546 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:44.529775 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:44.530207 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:44.530210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:44.530211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:44.530277 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:44.530573 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:44.530575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:44.530576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:44.530738 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:44.530861 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:44.531109 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 14:43:44.531206 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:43:44.535358 6034 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:44.536257 6034 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:44.541754 6034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f5d120d22054a44b3bd8d4072b91b37: Generated new TSK 0
I20260926 14:43:44.542591 6034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:44.546802 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 5952
W20260926 14:43:44.551609 5936 connection.cc:535] client connection to 127.16.90.254:39301 recv error: Network error: failed to read from TLS socket (remote: 127.16.90.254:39301): Connection reset by peer (error 104)
2026-09-26T14:43:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:44 dist-test-slave-x4sn krb5kdc[6051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:44 dist-test-slave-x4sn krb5kdc[6051](info): set up 2 sockets
Sep 26 14:43:44 dist-test-slave-x4sn krb5kdc[6051](info): commencing operation
krb5kdc: starting...
W20260926 14:43:45.505434 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:45 dist-test-slave-x4sn krb5kdc[6051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:43:45.616132 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:43237
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:43237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:45.719096 6064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:45.719345 6064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:45.719384 6064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:45.723289 6064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:45.723346 6064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:45.723366 6064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:45.723387 6064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:45.723412 6064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:45.728523 6064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:43237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:43237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:45.729476 6064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:45.730402 6064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:45.738337 6072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:45.738339 6069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:45.738368 6070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:45.738430 6064 server_base.cc:1034] running on GCE node
I20260926 14:43:45.739117 6064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:45.740135 6064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:45.741283 6064 hybrid_clock.cc:648] HybridClock initialized: now 1790433825741261 us; error 36 us; skew 500 ppm
Sep 26 14:43:45 dist-test-slave-x4sn krb5kdc[6051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433825, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:45.747689 6064 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:43:45.748590 6064 webserver.cc:460] Webserver started at http://127.16.90.254:43669/ using document root <none> and password file <none>
I20260926 14:43:45.749114 6064 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:45.749173 6064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:45.749361 6064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:45.751360 6064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "b7c37384de5640b6adb7166db4be0c9c"
format_stamp: "Formatted at 2026-09-26 14:43:45 on dist-test-slave-x4sn"
server_key: "d61d947a0e0c75c16e80e04b77c7540b"
server_key_iv: "1d91fddd7250827d85ecf869266bf7d4"
server_key_version: "encryptionkey@0"
I20260926 14:43:45.751889 6064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "b7c37384de5640b6adb7166db4be0c9c"
format_stamp: "Formatted at 2026-09-26 14:43:45 on dist-test-slave-x4sn"
server_key: "d61d947a0e0c75c16e80e04b77c7540b"
server_key_iv: "1d91fddd7250827d85ecf869266bf7d4"
server_key_version: "encryptionkey@0"
I20260926 14:43:45.755324 6064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:43:45.757689 6079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:45.758947 6064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:45.759056 6064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "b7c37384de5640b6adb7166db4be0c9c"
format_stamp: "Formatted at 2026-09-26 14:43:45 on dist-test-slave-x4sn"
server_key: "d61d947a0e0c75c16e80e04b77c7540b"
server_key_iv: "1d91fddd7250827d85ecf869266bf7d4"
server_key_version: "encryptionkey@0"
I20260926 14:43:45.759146 6064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:45.771845 6064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:45.774531 6064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:45.774701 6064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:45.781273 6064 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:43237
I20260926 14:43:45.781291 6132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:43237 every 8 connection(s)
I20260926 14:43:45.782143 6064 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:45.784883 6133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:45.790285 6133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c: Bootstrap starting.
I20260926 14:43:45.791127 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 6064
I20260926 14:43:45.791255 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:45.791481 16747 external_mini_cluster.cc:1428] Setting key fc37be5024265feb44aaca615ded7e21
I20260926 14:43:45.793062 6133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:45.793795 6133 log.cc:824] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:45.795763 6133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c: No bootstrap required, opened a new log
I20260926 14:43:45.799404 6133 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c37384de5640b6adb7166db4be0c9c" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43237 } }
I20260926 14:43:45.799636 6133 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:45.799723 6133 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7c37384de5640b6adb7166db4be0c9c, State: Initialized, Role: FOLLOWER
I20260926 14:43:45.800031 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:45.792614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40825 (local address 127.16.90.254:43237)
0926 14:43:45.793047 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:45.793063 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:45.793097 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:43:45.793752 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:45.793769 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:45.794125 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:45.794427 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:45.794826 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:45.795896 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:45.796792 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:45.797267 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:45.797417 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:45.798307 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:45.798312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:45.798326 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:45.798707 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:45.798760 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:45.798780 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:45.798891 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:45.799076 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 14:43:45.799203 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
I20260926 14:43:45.800169 6133 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c37384de5640b6adb7166db4be0c9c" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43237 } }
I20260926 14:43:45.800325 6133 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:45.800400 6133 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:45.800503 6133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:45.801386 6133 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c37384de5640b6adb7166db4be0c9c" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43237 } }
I20260926 14:43:45.801729 6133 leader_election.cc:302] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7c37384de5640b6adb7166db4be0c9c; no voters:
I20260926 14:43:45.802086 6133 leader_election.cc:288] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:45.802153 6138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:45.802395 6138 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [term 1 LEADER]: Becoming Leader. State: Replica: b7c37384de5640b6adb7166db4be0c9c, State: Running, Role: LEADER
I20260926 14:43:45.802737 6138 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c37384de5640b6adb7166db4be0c9c" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43237 } }
I20260926 14:43:45.802873 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:45.803326 6133 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:45.804334 6139 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7c37384de5640b6adb7166db4be0c9c. Latest consensus state: current_term: 1 leader_uuid: "b7c37384de5640b6adb7166db4be0c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c37384de5640b6adb7166db4be0c9c" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43237 } } }
I20260926 14:43:45.804450 6139 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:45.804746 6140 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7c37384de5640b6adb7166db4be0c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c37384de5640b6adb7166db4be0c9c" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43237 } } }
I20260926 14:43:45.804819 6140 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:45.805687 6151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:45.808495 6151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:45.813737 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:45.808754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44813 (local address 127.16.90.254:43237)
0926 14:43:45.808857 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:45.808861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:45.809300 (+ 439us) server_negotiation.cc:407] Connection header received
0926 14:43:45.809428 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:45.809431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:45.809477 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:45.809562 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:45.809868 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:45.810651 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:45.811842 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:45.812199 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:45.812299 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:45.813236 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:45.813240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:45.813243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:45.813256 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:45.813283 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:45.813296 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:45.813388 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:45.813559 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 14:43:45.813594 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 14:43:45.814456 6151 catalog_manager.cc:1347] Generated new cluster ID: d1ccd51b5048438c95c64905660d287d
I20260926 14:43:45.814514 6151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:45.820677 6151 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:45.821575 6151 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:45.826574 6151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7c37384de5640b6adb7166db4be0c9c: Generated new TSK 0
I20260926 14:43:45.827221 6151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:45.835239 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 6064
W20260926 14:43:45.840176 6049 connection.cc:535] client connection to 127.16.90.254:43237 recv error: Network error: failed to read from TLS socket (remote: 127.16.90.254:43237): Connection reset by peer (error 104)
2026-09-26T14:43:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:45 dist-test-slave-x4sn krb5kdc[6164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:45 dist-test-slave-x4sn krb5kdc[6164](info): set up 2 sockets
Sep 26 14:43:45 dist-test-slave-x4sn krb5kdc[6164](info): commencing operation
krb5kdc: starting...
W20260926 14:43:46.781207 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:46 dist-test-slave-x4sn krb5kdc[6164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:43:46.894287 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:39111
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:39111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:47.001484 6177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:47.001771 6177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:47.001812 6177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:47.005793 6177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:47.005858 6177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:47.005877 6177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:47.005892 6177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:47.005905 6177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:47.011152 6177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:39111
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:39111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:47.012274 6177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:47.013288 6177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:47.021230 6185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:47.021345 6183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:47.021230 6182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:47.021477 6177 server_base.cc:1034] running on GCE node
I20260926 14:43:47.022082 6177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:47.023159 6177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:47.024318 6177 hybrid_clock.cc:648] HybridClock initialized: now 1790433827024294 us; error 37 us; skew 500 ppm
Sep 26 14:43:47 dist-test-slave-x4sn krb5kdc[6164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433827, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:47.026962 6177 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:43:47.027876 6177 webserver.cc:460] Webserver started at http://127.16.90.254:40365/ using document root <none> and password file <none>
I20260926 14:43:47.028411 6177 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:47.028486 6177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:47.028684 6177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:47.030711 6177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "4d05416318074b92a1f538f86ebf54be"
format_stamp: "Formatted at 2026-09-26 14:43:47 on dist-test-slave-x4sn"
server_key: "2920f4d00c08fd2f9e60f68517880247"
server_key_iv: "3affacae4dab6871bde9d620aa9e7b98"
server_key_version: "encryptionkey@0"
I20260926 14:43:47.031247 6177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "4d05416318074b92a1f538f86ebf54be"
format_stamp: "Formatted at 2026-09-26 14:43:47 on dist-test-slave-x4sn"
server_key: "2920f4d00c08fd2f9e60f68517880247"
server_key_iv: "3affacae4dab6871bde9d620aa9e7b98"
server_key_version: "encryptionkey@0"
I20260926 14:43:47.034832 6177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 14:43:47.037146 6192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:47.038339 6177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:43:47.038451 6177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "4d05416318074b92a1f538f86ebf54be"
format_stamp: "Formatted at 2026-09-26 14:43:47 on dist-test-slave-x4sn"
server_key: "2920f4d00c08fd2f9e60f68517880247"
server_key_iv: "3affacae4dab6871bde9d620aa9e7b98"
server_key_version: "encryptionkey@0"
I20260926 14:43:47.038542 6177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:47.043025 6177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:47.046041 6177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:47.046202 6177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:47.052860 6177 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:39111
I20260926 14:43:47.052911 6245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:39111 every 8 connection(s)
I20260926 14:43:47.053746 6177 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:47.056373 6246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:47.059544 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 6177
I20260926 14:43:47.059698 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:47.059952 16747 external_mini_cluster.cc:1428] Setting key 030adefa2622d705b44adcaf3da2286d
I20260926 14:43:47.061944 6246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be: Bootstrap starting.
I20260926 14:43:47.064414 6246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:47.065037 6249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:47.061326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50197 (local address 127.16.90.254:39111)
0926 14:43:47.061720 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:47.061735 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:47.061772 (+ 37us) server_negotiation.cc:407] Connection header received
0926 14:43:47.062418 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:47.062435 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:47.062722 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:47.062992 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:47.063807 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:47.063811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:47.063824 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:47.063834 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:47.063888 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:47.063907 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:47.063989 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:47.064111 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 14:43:47.064253 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":96,"threads_started":1}
I20260926 14:43:47.065140 6246 log.cc:824] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:47.067176 6246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be: No bootstrap required, opened a new log
I20260926 14:43:47.067744 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:47.071280 6249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:47.069924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38937 (local address 127.16.90.254:39111)
0926 14:43:47.070032 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:47.070037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:47.070155 (+ 118us) server_negotiation.cc:407] Connection header received
0926 14:43:47.070240 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:47.070243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:47.070279 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:47.070368 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:47.070945 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:47.070949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:47.070951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:47.070961 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:47.070988 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:47.070999 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:47.071087 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:47.071140 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 14:43:47.071174 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:43:47.071487 6246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d05416318074b92a1f538f86ebf54be" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39111 } }
I20260926 14:43:47.071791 6246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:47.071849 6246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d05416318074b92a1f538f86ebf54be, State: Initialized, Role: FOLLOWER
I20260926 14:43:47.072299 6246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d05416318074b92a1f538f86ebf54be" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39111 } }
I20260926 14:43:47.072436 6246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:47.072481 6246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:47.072549 6246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:47.073357 6246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d05416318074b92a1f538f86ebf54be" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39111 } }
I20260926 14:43:47.073652 6246 leader_election.cc:302] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d05416318074b92a1f538f86ebf54be; no voters:
I20260926 14:43:47.073886 6246 leader_election.cc:288] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:47.073998 6256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:47.074225 6256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [term 1 LEADER]: Becoming Leader. State: Replica: 4d05416318074b92a1f538f86ebf54be, State: Running, Role: LEADER
I20260926 14:43:47.074522 6256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d05416318074b92a1f538f86ebf54be" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39111 } }
I20260926 14:43:47.075035 6246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:47.076143 6258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d05416318074b92a1f538f86ebf54be. Latest consensus state: current_term: 1 leader_uuid: "4d05416318074b92a1f538f86ebf54be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d05416318074b92a1f538f86ebf54be" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39111 } } }
I20260926 14:43:47.076143 6257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d05416318074b92a1f538f86ebf54be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d05416318074b92a1f538f86ebf54be" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 39111 } } }
I20260926 14:43:47.076351 6257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:47.076376 6258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:47.076714 6265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:47.079695 6265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:47.084998 6265 catalog_manager.cc:1347] Generated new cluster ID: 1f6900e94bd2407db11d5286c1af2248
I20260926 14:43:47.085058 6265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:47.103412 6265 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:47.103794 6265 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:47.113969 6265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d05416318074b92a1f538f86ebf54be: Generated new TSK 0
I20260926 14:43:47.114794 6265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:47.127388 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 6177
2026-09-26T14:43:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:47 dist-test-slave-x4sn krb5kdc[6277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:47 dist-test-slave-x4sn krb5kdc[6277](info): set up 2 sockets
Sep 26 14:43:47 dist-test-slave-x4sn krb5kdc[6277](info): commencing operation
krb5kdc: starting...
W20260926 14:43:48.077250 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:48 dist-test-slave-x4sn krb5kdc[6277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:43:48.190451 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:33527
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:33527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:48.293879 6290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:48.294129 6290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:48.294185 6290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:48.298038 6290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:48.298094 6290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:48.298120 6290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:48.298139 6290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:48.298161 6290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:48.303215 6290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:33527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:33527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:48.304203 6290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:48.305198 6290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:48.313374 6296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:48.313418 6298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:48.313401 6295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:48.313611 6290 server_base.cc:1034] running on GCE node
I20260926 14:43:48.314165 6290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:48.315217 6290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:48.316362 6290 hybrid_clock.cc:648] HybridClock initialized: now 1790433828316336 us; error 37 us; skew 500 ppm
Sep 26 14:43:48 dist-test-slave-x4sn krb5kdc[6277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433828, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:48.319005 6290 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:43:48.319919 6290 webserver.cc:460] Webserver started at http://127.16.90.254:46681/ using document root <none> and password file <none>
I20260926 14:43:48.320468 6290 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:48.320533 6290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:48.320724 6290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:48.322721 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "60b7c48f94504f0e96338b8ce177de8a"
format_stamp: "Formatted at 2026-09-26 14:43:48 on dist-test-slave-x4sn"
server_key: "4af75e64edbbba2f1e86e2dda9cc4357"
server_key_iv: "23719818baec9b7e793b2096312535a8"
server_key_version: "encryptionkey@0"
I20260926 14:43:48.323246 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "60b7c48f94504f0e96338b8ce177de8a"
format_stamp: "Formatted at 2026-09-26 14:43:48 on dist-test-slave-x4sn"
server_key: "4af75e64edbbba2f1e86e2dda9cc4357"
server_key_iv: "23719818baec9b7e793b2096312535a8"
server_key_version: "encryptionkey@0"
I20260926 14:43:48.326823 6290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:43:48.329393 6305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:48.330699 6290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:48.330806 6290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "60b7c48f94504f0e96338b8ce177de8a"
format_stamp: "Formatted at 2026-09-26 14:43:48 on dist-test-slave-x4sn"
server_key: "4af75e64edbbba2f1e86e2dda9cc4357"
server_key_iv: "23719818baec9b7e793b2096312535a8"
server_key_version: "encryptionkey@0"
I20260926 14:43:48.330899 6290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:48.355090 6290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:48.358212 6290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:48.358384 6290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:48.365020 6290 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:33527
I20260926 14:43:48.365034 6358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:33527 every 8 connection(s)
I20260926 14:43:48.365931 6290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:48.368594 6359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:48.373725 6359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a: Bootstrap starting.
I20260926 14:43:48.375627 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 6290
I20260926 14:43:48.375802 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:48.376065 16747 external_mini_cluster.cc:1428] Setting key 60dd744ec791900534acc8f783e6697d
I20260926 14:43:48.376286 6359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:48.377099 6359 log.cc:824] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:48.378994 6359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a: No bootstrap required, opened a new log
I20260926 14:43:48.382954 6359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b7c48f94504f0e96338b8ce177de8a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33527 } }
I20260926 14:43:48.383235 6359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:48.383294 6359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b7c48f94504f0e96338b8ce177de8a, State: Initialized, Role: FOLLOWER
I20260926 14:43:48.383803 6359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b7c48f94504f0e96338b8ce177de8a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33527 } }
I20260926 14:43:48.383956 6359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:48.384021 6359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:48.384120 6359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [term 0 FOLLOWER]: Advancing to term 1
Sep 26 14:43:48 dist-test-slave-x4sn krb5kdc[6277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:48.385077 6359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b7c48f94504f0e96338b8ce177de8a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33527 } }
I20260926 14:43:48.385401 6359 leader_election.cc:302] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60b7c48f94504f0e96338b8ce177de8a; no voters:
I20260926 14:43:48.385660 6359 leader_election.cc:288] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:48.385865 6364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:48.386318 6364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [term 1 LEADER]: Becoming Leader. State: Replica: 60b7c48f94504f0e96338b8ce177de8a, State: Running, Role: LEADER
I20260926 14:43:48.386647 6364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b7c48f94504f0e96338b8ce177de8a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33527 } }
I20260926 14:43:48.386945 6359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:48.388233 6365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60b7c48f94504f0e96338b8ce177de8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b7c48f94504f0e96338b8ce177de8a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33527 } } }
I20260926 14:43:48.388355 6365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:48.388837 6374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:48.389317 6366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60b7c48f94504f0e96338b8ce177de8a. Latest consensus state: current_term: 1 leader_uuid: "60b7c48f94504f0e96338b8ce177de8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b7c48f94504f0e96338b8ce177de8a" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 33527 } } }
I20260926 14:43:48.389421 6366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:48.391072 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:48.377137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47745 (local address 127.16.90.254:33527)
0926 14:43:48.377544 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:48.377560 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:48.377596 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:43:48.378231 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:48.378247 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:48.378570 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:48.378841 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:48.379696 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:48.380686 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:48.381883 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:48.382350 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:48.382464 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:48.385167 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:48.385174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:48.385191 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:48.385225 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:48.387165 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:48.387649 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:48.387655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:48.387692 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:48.387849 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:48.388231 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:48.388235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:48.388236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:48.388756 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:48.388945 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:48.389826 (+ 881us) server_negotiation.cc:299] Negotiation successful
0926 14:43:48.390139 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":120,"threads_started":1}
I20260926 14:43:48.392120 6374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:48.394084 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:48.397926 6374 catalog_manager.cc:1347] Generated new cluster ID: 49cb151de5f444d8aba5fba076eb48a8
I20260926 14:43:48.398017 6374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:48.403350 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:48.396174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51679 (local address 127.16.90.254:33527)
0926 14:43:48.396277 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:48.396281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:48.396455 (+ 174us) server_negotiation.cc:407] Connection header received
0926 14:43:48.396585 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:48.396588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:48.396636 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:48.396715 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:48.397514 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:48.398227 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:48.399034 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:48.399291 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:48.399376 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:48.400625 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:48.400628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:48.400631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:48.400656 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:48.401907 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:48.402278 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:48.402281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:48.402283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:48.402358 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:48.402595 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:48.402598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:48.402600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:48.402769 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:48.402883 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:48.403105 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 14:43:48.403211 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 14:43:48.442673 6374 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:48.443890 6374 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:48.458063 6374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60b7c48f94504f0e96338b8ce177de8a: Generated new TSK 0
I20260926 14:43:48.458762 6374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:48.471215 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 6290
2026-09-26T14:43:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:48 dist-test-slave-x4sn krb5kdc[6390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:48 dist-test-slave-x4sn krb5kdc[6390](info): set up 2 sockets
Sep 26 14:43:48 dist-test-slave-x4sn krb5kdc[6390](info): commencing operation
krb5kdc: starting...
W20260926 14:43:49.425392 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:49 dist-test-slave-x4sn krb5kdc[6390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:43:49.542603 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35827
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:45541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:35827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:49.652324 6403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:49.652585 6403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:49.652626 6403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:49.656576 6403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:49.656635 6403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:49.656653 6403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:49.656668 6403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:49.656682 6403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:49.661890 6403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:45541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:35827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:35827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:49.662892 6403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:49.663866 6403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:49.672044 6409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:49.672027 6408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:49.672209 6403 server_base.cc:1034] running on GCE node
W20260926 14:43:49.672016 6411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:49.672850 6403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:49.673935 6403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:49.675149 6403 hybrid_clock.cc:648] HybridClock initialized: now 1790433829675123 us; error 37 us; skew 500 ppm
Sep 26 14:43:49 dist-test-slave-x4sn krb5kdc[6390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433829, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:49.677798 6403 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:43:49.678661 6403 webserver.cc:460] Webserver started at http://127.16.90.254:34589/ using document root <none> and password file <none>
I20260926 14:43:49.679159 6403 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:49.679219 6403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:49.679376 6403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:49.681397 6403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "642ea452409a440fae8585b367afb482"
format_stamp: "Formatted at 2026-09-26 14:43:49 on dist-test-slave-x4sn"
server_key: "043ba2fabf3dcb5a95b18965ada36485"
server_key_iv: "94ec288c875fa86d23e10e031663e76a"
server_key_version: "encryptionkey@0"
I20260926 14:43:49.681922 6403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "642ea452409a440fae8585b367afb482"
format_stamp: "Formatted at 2026-09-26 14:43:49 on dist-test-slave-x4sn"
server_key: "043ba2fabf3dcb5a95b18965ada36485"
server_key_iv: "94ec288c875fa86d23e10e031663e76a"
server_key_version: "encryptionkey@0"
I20260926 14:43:49.685415 6403 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:43:49.687636 6418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:49.688867 6403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:49.688979 6403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "642ea452409a440fae8585b367afb482"
format_stamp: "Formatted at 2026-09-26 14:43:49 on dist-test-slave-x4sn"
server_key: "043ba2fabf3dcb5a95b18965ada36485"
server_key_iv: "94ec288c875fa86d23e10e031663e76a"
server_key_version: "encryptionkey@0"
I20260926 14:43:49.689071 6403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:49.706861 6403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:49.709899 6403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:49.710079 6403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:49.716755 6403 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:35827
I20260926 14:43:49.716780 6470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:35827 every 8 connection(s)
I20260926 14:43:49.717677 6403 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:49.718644 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 6403
I20260926 14:43:49.718755 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:49.718997 16747 external_mini_cluster.cc:1428] Setting key 2e1188d09517e170bf9ba34f87894eaf
I20260926 14:43:49.720597 6471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:49.727104 6471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482: Bootstrap starting.
I20260926 14:43:49.728724 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:49.720369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53173 (local address 127.16.90.254:35827)
0926 14:43:49.720790 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:49.720807 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:49.720849 (+ 42us) server_negotiation.cc:407] Connection header received
0926 14:43:49.721691 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:49.721725 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:49.722135 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:49.722481 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:49.722873 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:49.724346 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:49.725258 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:49.725940 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:49.726096 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:49.726805 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:49.726811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:49.726826 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:49.727288 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:49.727351 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:49.727379 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:49.727470 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:49.727583 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 14:43:49.727808 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":127,"threads_started":1}
I20260926 14:43:49.729735 6471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:49.730479 6471 log.cc:824] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:49.731374 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:49.732417 6471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482: No bootstrap required, opened a new log
I20260926 14:43:49.736444 6471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642ea452409a440fae8585b367afb482" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35827 } }
I20260926 14:43:49.736660 6471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:49.736713 6471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642ea452409a440fae8585b367afb482, State: Initialized, Role: FOLLOWER
I20260926 14:43:49.737196 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:49.733299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43559 (local address 127.16.90.254:35827)
0926 14:43:49.733413 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:49.733418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:49.733617 (+ 199us) server_negotiation.cc:407] Connection header received
0926 14:43:49.733692 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:49.733695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:49.733735 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:49.733811 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:49.734028 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:49.734779 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:49.735642 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:49.736006 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:49.736111 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:49.736796 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:49.736801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:49.736803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:49.736816 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:49.736841 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:49.736853 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:49.736919 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:49.737028 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 14:43:49.737077 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:43:49.737206 6471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642ea452409a440fae8585b367afb482" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35827 } }
I20260926 14:43:49.737398 6471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:49.737459 6471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:49.737550 6471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:49.738453 6471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642ea452409a440fae8585b367afb482" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35827 } }
I20260926 14:43:49.738759 6471 leader_election.cc:302] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 642ea452409a440fae8585b367afb482; no voters:
I20260926 14:43:49.739018 6471 leader_election.cc:288] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:49.739161 6481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:49.739358 6481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [term 1 LEADER]: Becoming Leader. State: Replica: 642ea452409a440fae8585b367afb482, State: Running, Role: LEADER
I20260926 14:43:49.739658 6481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642ea452409a440fae8585b367afb482" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35827 } }
I20260926 14:43:49.740340 6471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:49.740772 6481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 642ea452409a440fae8585b367afb482. Latest consensus state: current_term: 1 leader_uuid: "642ea452409a440fae8585b367afb482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642ea452409a440fae8585b367afb482" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35827 } } }
I20260926 14:43:49.740965 6481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:49.741153 6482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "642ea452409a440fae8585b367afb482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642ea452409a440fae8585b367afb482" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 35827 } } }
I20260926 14:43:49.741319 6482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:49.741402 6488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:49.744547 6488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:49.750090 6488 catalog_manager.cc:1347] Generated new cluster ID: 13645d5c68634e8aaf1844c1374a40ea
I20260926 14:43:49.750156 6488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:49.756366 6488 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:49.757547 6488 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:49.771517 6488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 642ea452409a440fae8585b367afb482: Generated new TSK 0
I20260926 14:43:49.772405 6488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:49.777295 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 6403
2026-09-26T14:43:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:49 dist-test-slave-x4sn krb5kdc[6503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:49 dist-test-slave-x4sn krb5kdc[6503](info): set up 2 sockets
Sep 26 14:43:49 dist-test-slave-x4sn krb5kdc[6503](info): commencing operation
krb5kdc: starting...
W20260926 14:43:50.729211 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:50 dist-test-slave-x4sn krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:43:50.843051 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:40923
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:40923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:50.950317 6516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:50.950598 6516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:50.950654 6516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:50.954624 6516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:50.954695 6516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:50.954722 6516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:50.954743 6516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:50.954763 6516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:50.960078 6516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:40923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:40923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:50.961120 6516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:50.962134 6516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:50.970145 6521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:50.970228 6516 server_base.cc:1034] running on GCE node
W20260926 14:43:50.970232 6524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:50.970147 6522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:50.970914 6516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:50.971974 6516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:50.973121 6516 hybrid_clock.cc:648] HybridClock initialized: now 1790433830973108 us; error 26 us; skew 500 ppm
Sep 26 14:43:50 dist-test-slave-x4sn krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433830, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:50.980010 6516 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:43:50.980870 6516 webserver.cc:460] Webserver started at http://127.16.90.254:41799/ using document root <none> and password file <none>
I20260926 14:43:50.981389 6516 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:50.981454 6516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:50.981631 6516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:50.983624 6516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "8d61e931145a41919f5b43b574c7bf6f"
format_stamp: "Formatted at 2026-09-26 14:43:50 on dist-test-slave-x4sn"
server_key: "bf8cde7f911619942489f1c024685e0a"
server_key_iv: "5fac3ddb23777a1c4dffefbccbf94fc7"
server_key_version: "encryptionkey@0"
I20260926 14:43:50.984176 6516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "8d61e931145a41919f5b43b574c7bf6f"
format_stamp: "Formatted at 2026-09-26 14:43:50 on dist-test-slave-x4sn"
server_key: "bf8cde7f911619942489f1c024685e0a"
server_key_iv: "5fac3ddb23777a1c4dffefbccbf94fc7"
server_key_version: "encryptionkey@0"
I20260926 14:43:50.987540 6516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:43:50.989800 6531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:50.990938 6516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:50.991043 6516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "8d61e931145a41919f5b43b574c7bf6f"
format_stamp: "Formatted at 2026-09-26 14:43:50 on dist-test-slave-x4sn"
server_key: "bf8cde7f911619942489f1c024685e0a"
server_key_iv: "5fac3ddb23777a1c4dffefbccbf94fc7"
server_key_version: "encryptionkey@0"
I20260926 14:43:50.991127 6516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:50.995409 6516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:50.998237 6516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:50.998395 6516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:51.005077 6516 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:40923
I20260926 14:43:51.005105 6583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:40923 every 8 connection(s)
I20260926 14:43:51.005970 6516 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:51.008272 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 6516
I20260926 14:43:51.008417 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:51.008746 16747 external_mini_cluster.cc:1428] Setting key 95a6f455bb3c33be0ea3dbea0e427420
I20260926 14:43:51.009078 6584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:51.013756 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:51.010152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54123 (local address 127.16.90.254:40923)
0926 14:43:51.010616 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:51.010631 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:51.010666 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:43:51.011290 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:51.011305 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:51.011571 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:51.011896 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:51.012636 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:51.012640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:51.012651 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:51.012662 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:51.012716 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:51.012735 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:51.012855 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:51.012946 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 14:43:51.013093 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":175,"threads_started":1}
I20260926 14:43:51.015755 6584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f: Bootstrap starting.
I20260926 14:43:51.016232 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:51.018049 6584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:51.019011 6584 log.cc:824] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:51.019567 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:51.018044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42887 (local address 127.16.90.254:40923)
0926 14:43:51.018145 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:51.018149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:51.018317 (+ 168us) server_negotiation.cc:407] Connection header received
0926 14:43:51.018456 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:51.018458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:51.018483 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:51.018540 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:51.019198 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:51.019201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:51.019204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:43:51.019215 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:51.019241 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:43:51.019251 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:51.019313 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:51.019426 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 14:43:51.019460 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:43:51.021095 6584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f: No bootstrap required, opened a new log
I20260926 14:43:51.024358 6584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d61e931145a41919f5b43b574c7bf6f" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40923 } }
I20260926 14:43:51.024559 6584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:43:51.024603 6584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d61e931145a41919f5b43b574c7bf6f, State: Initialized, Role: FOLLOWER
I20260926 14:43:51.025007 6584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d61e931145a41919f5b43b574c7bf6f" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40923 } }
I20260926 14:43:51.025141 6584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:43:51.025225 6584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:43:51.025322 6584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:51.026131 6584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d61e931145a41919f5b43b574c7bf6f" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40923 } }
I20260926 14:43:51.026436 6584 leader_election.cc:302] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d61e931145a41919f5b43b574c7bf6f; no voters:
I20260926 14:43:51.026715 6584 leader_election.cc:288] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:43:51.026806 6594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:51.027025 6594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [term 1 LEADER]: Becoming Leader. State: Replica: 8d61e931145a41919f5b43b574c7bf6f, State: Running, Role: LEADER
I20260926 14:43:51.027334 6594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d61e931145a41919f5b43b574c7bf6f" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40923 } }
I20260926 14:43:51.027830 6584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:51.028821 6595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d61e931145a41919f5b43b574c7bf6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d61e931145a41919f5b43b574c7bf6f" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40923 } } }
I20260926 14:43:51.028946 6595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:51.029300 6603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:51.032090 6603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:51.029286 6596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d61e931145a41919f5b43b574c7bf6f. Latest consensus state: current_term: 1 leader_uuid: "8d61e931145a41919f5b43b574c7bf6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d61e931145a41919f5b43b574c7bf6f" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 40923 } } }
I20260926 14:43:51.034192 6596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:51.037192 6603 catalog_manager.cc:1347] Generated new cluster ID: c5c766ff6fca43ad8caeae03fc4d9cd9
I20260926 14:43:51.037251 6603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:51.048954 6603 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:51.049225 6603 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:51.052773 6603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d61e931145a41919f5b43b574c7bf6f: Generated new TSK 0
I20260926 14:43:51.053382 6603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:43:51.065253 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 6516
2026-09-26T14:43:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1287 ms)
[----------] 18 tests from AuthTokenIssuingTest (23343 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:51 dist-test-slave-x4sn krb5kdc[6615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:51 dist-test-slave-x4sn krb5kdc[6615](info): set up 2 sockets
Sep 26 14:43:51 dist-test-slave-x4sn krb5kdc[6615](info): commencing operation
krb5kdc: starting...
W20260926 14:43:52.001299 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:43:52.112720 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:43557
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:33325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:43557,127.16.90.253:32823,127.16.90.252:46389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:52.217190 6628 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:43:52.217459 6628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:52.217480 6628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:52.217523 6628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:52.221375 6628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:52.221426 6628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:52.221443 6628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:52.221458 6628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:52.221477 6628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:52.226512 6628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:33325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:43557,127.16.90.253:32823,127.16.90.252:46389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:43557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:52.227555 6628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:52.228567 6628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:52.236343 6634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:52.236351 6633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:52.236402 6636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:52.236753 6628 server_base.cc:1034] running on GCE node
I20260926 14:43:52.237290 6628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:52.238314 6628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:52.239470 6628 hybrid_clock.cc:648] HybridClock initialized: now 1790433832239460 us; error 27 us; skew 500 ppm
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:52.241999 6628 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:43:52.242799 6628 webserver.cc:460] Webserver started at http://127.16.90.254:35329/ using document root <none> and password file <none>
I20260926 14:43:52.243310 6628 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:52.243372 6628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:52.243546 6628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:52.245492 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "aa843094f3f741e592ac3ed8ccb4f4a9"
format_stamp: "Formatted at 2026-09-26 14:43:52 on dist-test-slave-x4sn"
server_key: "9acbc0e0caeeb94faa2681dc4363c1a2"
server_key_iv: "e45a5d2476052e43aa1e1589fb5c5740"
server_key_version: "encryptionkey@0"
I20260926 14:43:52.246004 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "aa843094f3f741e592ac3ed8ccb4f4a9"
format_stamp: "Formatted at 2026-09-26 14:43:52 on dist-test-slave-x4sn"
server_key: "9acbc0e0caeeb94faa2681dc4363c1a2"
server_key_iv: "e45a5d2476052e43aa1e1589fb5c5740"
server_key_version: "encryptionkey@0"
I20260926 14:43:52.249434 6628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:43:52.251605 6643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:52.252772 6628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:52.252879 6628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "aa843094f3f741e592ac3ed8ccb4f4a9"
format_stamp: "Formatted at 2026-09-26 14:43:52 on dist-test-slave-x4sn"
server_key: "9acbc0e0caeeb94faa2681dc4363c1a2"
server_key_iv: "e45a5d2476052e43aa1e1589fb5c5740"
server_key_version: "encryptionkey@0"
I20260926 14:43:52.252972 6628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:52.265856 6628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:52.268471 6628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:52.268644 6628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:52.274986 6628 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:43557
I20260926 14:43:52.275238 6695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:43557 every 8 connection(s)
I20260926 14:43:52.275936 6628 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:52.277442 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 6628
I20260926 14:43:52.277554 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:52.277822 16747 external_mini_cluster.cc:1428] Setting key b0e1eacae0c49365800cabf66949eb88
I20260926 14:43:52.278589 6696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:52.282539 6696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.90.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.253@KRBTEST.COM" created.
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:52.297326 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.284862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:43557 (local address 127.0.0.1:60082)
0926 14:43:52.285354 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.285394 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:52.286206 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:52.286860 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:52.286864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:52.287250 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:52.287584 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.287599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.289078 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.289081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.290059 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.290068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.290855 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.290858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.290961 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.291463 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:52.291478 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:52.292833 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:52.294813 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.294820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.294827 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.295025 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.295324 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.295326 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.295327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.295424 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.295968 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:52.295973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:52.296122 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 14:43:52.296307 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 14:43:52.296618 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":62,"thread_start_us":140,"threads_started":1}
I20260926 14:43:52.297333 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.285646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.16.90.254:43557)
0926 14:43:52.285946 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.285955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.285991 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:43:52.286456 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.286460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.286695 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.286841 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.287754 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.288962 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.290170 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.290723 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.290936 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.292973 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.292979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.292993 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.293018 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.294704 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.295122 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.295127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.295128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.295225 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.295488 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.295490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.295491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.295881 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.295980 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.296363 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.296567 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
I20260926 14:43:52.299611 6696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.253.keytab.
Entry for principal kudu/127.16.90.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.253.keytab.
I20260926 14:43:52.300328 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.300107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:32823 (local address 127.0.0.1:35460)
0926 14:43:52.300213 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.300276 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 14:43:52.300562 6647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.90.253:32823: connect: Connection refused (error 111)
I20260926 14:43:52.300840 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.253:32823
--webserver_interface=127.16.90.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.253.keytab
--principal=kudu/127.16.90.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:33325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:43557,127.16.90.253:32823,127.16.90.252:46389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:43:52.301311 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.301124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:32823 (local address 127.0.0.1:35464)
0926 14:43:52.301222 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.301266 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 14:43:52.301421 6696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.253:32823: Network error: Client connection negotiation failed: client connection to 127.16.90.253:32823: connect: Connection refused (error 111)
I20260926 14:43:52.350641 6696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } attempt: 1
I20260926 14:43:52.351504 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.351225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:32823 (local address 127.0.0.1:35478)
0926 14:43:52.351364 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.351426 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 14:43:52.352192 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.351976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:32823 (local address 127.0.0.1:35488)
0926 14:43:52.352075 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.352117 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:43:52.352303 6696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.253:32823: Network error: Client connection negotiation failed: client connection to 127.16.90.253:32823: connect: Connection refused (error 111)
W20260926 14:43:52.406956 6703 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:43:52.407232 6703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:52.407265 6703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:52.407327 6703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:52.411227 6703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:52.411283 6703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:52.411309 6703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:52.411331 6703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:52.411356 6703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:52.416409 6703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:33325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:43557,127.16.90.253:32823,127.16.90.252:46389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.253.keytab
--principal=kudu/127.16.90.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.253:32823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.90.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:52.417383 6703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:52.418335 6703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:43:52.420436 6696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } attempt: 2
I20260926 14:43:52.421207 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.420939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:32823 (local address 127.0.0.1:35500)
0926 14:43:52.421071 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.421129 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 14:43:52.421837 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.421632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:32823 (local address 127.0.0.1:35516)
0926 14:43:52.421737 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.421782 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 14:43:52.421945 6696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.253:32823: Network error: Client connection negotiation failed: client connection to 127.16.90.253:32823: connect: Connection refused (error 111)
W20260926 14:43:52.426517 6709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:52.426478 6711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:52.426469 6708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:52.426746 6703 server_base.cc:1034] running on GCE node
I20260926 14:43:52.427258 6703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:52.428301 6703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:52.429461 6703 hybrid_clock.cc:648] HybridClock initialized: now 1790433832429443 us; error 27 us; skew 500 ppm
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:52.432085 6703 init.cc:377] Logged in from keytab as kudu/127.16.90.253@KRBTEST.COM (short username kudu)
I20260926 14:43:52.433094 6703 webserver.cc:460] Webserver started at http://127.16.90.253:46335/ using document root <none> and password file <none>
I20260926 14:43:52.433666 6703 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:52.433734 6703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:52.433910 6703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:52.435946 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/data/instance:
uuid: "e4c344e7fc9948269cbbe6a875695ad0"
format_stamp: "Formatted at 2026-09-26 14:43:52 on dist-test-slave-x4sn"
server_key: "5f80425b46a299ee3defc09b691f6776"
server_key_iv: "fcc6be9be9d2cee25cca608ef66bdf7f"
server_key_version: "encryptionkey@0"
I20260926 14:43:52.436466 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/wal/instance:
uuid: "e4c344e7fc9948269cbbe6a875695ad0"
format_stamp: "Formatted at 2026-09-26 14:43:52 on dist-test-slave-x4sn"
server_key: "5f80425b46a299ee3defc09b691f6776"
server_key_iv: "fcc6be9be9d2cee25cca608ef66bdf7f"
server_key_version: "encryptionkey@0"
I20260926 14:43:52.440062 6703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:52.442291 6718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:52.443411 6703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:52.443542 6703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/wal
uuid: "e4c344e7fc9948269cbbe6a875695ad0"
format_stamp: "Formatted at 2026-09-26 14:43:52 on dist-test-slave-x4sn"
server_key: "5f80425b46a299ee3defc09b691f6776"
server_key_iv: "fcc6be9be9d2cee25cca608ef66bdf7f"
server_key_version: "encryptionkey@0"
I20260926 14:43:52.443642 6703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:52.464952 6703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:52.467656 6703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:52.467867 6703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:52.474510 6703 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.253:32823
I20260926 14:43:52.474742 6770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.253:32823 every 8 connection(s)
I20260926 14:43:52.475446 6703 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
I20260926 14:43:52.476454 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 6703
I20260926 14:43:52.476573 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-1/wal/instance
I20260926 14:43:52.476882 16747 external_mini_cluster.cc:1428] Setting key 75aa68716c88b3c417c5eab143354d5c
I20260926 14:43:52.478720 6771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:52.482406 6771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.90.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.252@KRBTEST.COM" created.
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.253@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:52.495748 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.484264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60098 (local address 127.16.90.254:43557)
0926 14:43:52.484402 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.484406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.485203 (+ 797us) server_negotiation.cc:407] Connection header received
0926 14:43:52.486038 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.486041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.486116 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.486196 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.487333 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.488145 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.489245 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.489599 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.489710 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.492268 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.492271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.492273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.492297 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.493708 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.494223 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.494226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.494227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.494335 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.494625 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.494628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.494629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.494867 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.494948 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.495489 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.495593 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:43:52.496558 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.484630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:43557 (local address 127.0.0.1:60098)
0926 14:43:52.485067 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.485108 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:52.485869 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:52.486383 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:52.486392 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:52.486707 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:52.487193 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.487205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.488272 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.488275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.489120 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.489130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.489736 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.489739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.489880 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.490445 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:52.490459 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:52.492099 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:52.493815 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.493819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.493830 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.494073 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.494422 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.494425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.494426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.494558 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.494975 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:52.494980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:52.495226 (+ 246us) client_negotiation.cc:769] Sending connection context
0926 14:43:52.495446 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 14:43:52.495750 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260926 14:43:52.497951 6771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.252.keytab.
Entry for principal kudu/127.16.90.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.252.keytab.
I20260926 14:43:52.500598 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.252:46389
--webserver_interface=127.16.90.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.252.keytab
--principal=kudu/127.16.90.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:33325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:43557,127.16.90.253:32823,127.16.90.252:46389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.253@KRBTEST.COM for kudu/127.16.90.253@KRBTEST.COM
I20260926 14:43:52.506906 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.498395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:32823 (local address 127.0.0.1:35526)
0926 14:43:52.498495 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.498508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:52.498572 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:52.499141 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:52.499144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:52.499328 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:52.499409 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.499415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.500653 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.500656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.501256 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.501265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.501996 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.501999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.502144 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.502544 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:52.502554 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:52.503731 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:52.505380 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.505383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.505386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.505623 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.505864 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.505867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.505869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.505959 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.506361 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:52.506364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:52.506467 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 14:43:52.506578 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 14:43:52.506712 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":38}
I20260926 14:43:52.506917 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.498483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.16.90.253:32823)
0926 14:43:52.498795 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.498804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.498821 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:43:52.498859 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.498864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.499032 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.499150 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.499504 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.500530 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.501362 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.501892 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.501988 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.503853 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.503862 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.503872 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.503892 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.505275 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.505716 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.505721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.505722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.505791 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.506039 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.506041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.506042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.506271 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.506367 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.506641 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.506764 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":238,"thread_start_us":148,"threads_started":1}
I20260926 14:43:52.508813 6771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:43:52.509479 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.509252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53642)
0926 14:43:52.509360 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.509424 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 14:43:52.509656 6720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
I20260926 14:43:52.510407 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.510232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53644)
0926 14:43:52.510324 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.510362 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 14:43:52.510522 6771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:46389: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
I20260926 14:43:52.513090 6696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } attempt: 3
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.253@KRBTEST.COM
I20260926 14:43:52.520836 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.513529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.16.90.253:32823)
0926 14:43:52.513635 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.513638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.513652 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:43:52.513777 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.513779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.513813 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.513873 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.514313 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.514820 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.515901 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.516136 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.516212 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.518204 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.518207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.518209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.518229 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.519440 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.519888 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.519892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.519894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.519963 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.520204 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.520206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.520207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.520355 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.520420 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.520651 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.520735 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:43:52.520905 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.513465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:32823 (local address 127.0.0.1:35534)
0926 14:43:52.513600 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.513613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:52.513699 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:52.513900 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:52.513903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:52.514085 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:52.514214 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.514220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.514903 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.514906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.515803 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.515813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.516241 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.516244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.516355 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.516774 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:52.516788 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:52.518084 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:52.519535 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.519538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.519540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.519802 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.520038 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.520041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.520043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.520125 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.520424 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:52.520426 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:52.520521 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 14:43:52.520621 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 14:43:52.520746 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 14:43:52.521883 6696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:43:52.522495 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.522294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53652)
0926 14:43:52.522399 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.522443 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 14:43:52.523128 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.522893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53654)
0926 14:43:52.523031 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.523072 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260926 14:43:52.523260 6696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:46389: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
I20260926 14:43:52.554436 6696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } attempt: 1
I20260926 14:43:52.555294 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.554984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53660)
0926 14:43:52.555157 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.555213 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 14:43:52.556102 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.555846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53662)
0926 14:43:52.555984 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.556034 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 14:43:52.556226 6696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:46389: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
I20260926 14:43:52.559754 6771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } attempt: 1
I20260926 14:43:52.560371 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.560140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53676)
0926 14:43:52.560256 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.560306 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 14:43:52.561026 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.560837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53686)
0926 14:43:52.560931 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.560976 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 14:43:52.561162 6771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:46389: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
W20260926 14:43:52.613363 6777 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:43:52.613637 6777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:52.613672 6777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:52.613734 6777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:52.617654 6777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:52.617712 6777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:52.617738 6777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:52.617758 6777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:52.617782 6777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:52.622865 6777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:33325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:43557,127.16.90.253:32823,127.16.90.252:46389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.252.keytab
--principal=kudu/127.16.90.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.252:46389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.90.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:52.623906 6777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:52.624871 6777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:43:52.629305 6771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } attempt: 2
I20260926 14:43:52.630062 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.629794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53692)
0926 14:43:52.629926 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.629992 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 14:43:52.630707 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.630503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53706)
0926 14:43:52.630606 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.630650 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:43:52.630810 6771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:46389: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
I20260926 14:43:52.631371 6696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } attempt: 2
I20260926 14:43:52.632295 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.631896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53708)
0926 14:43:52.632128 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.632212 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260926 14:43:52.633100 6783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:52.633328 6786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:52.633369 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.633100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53724)
0926 14:43:52.633256 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.633307 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 14:43:52.633420 6777 server_base.cc:1034] running on GCE node
W20260926 14:43:52.633126 6784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:52.633689 6696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:46389: Network error: Client connection negotiation failed: client connection to 127.16.90.252:46389: connect: Connection refused (error 111)
I20260926 14:43:52.634146 6777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:52.635174 6777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:52.636327 6777 hybrid_clock.cc:648] HybridClock initialized: now 1790433832636309 us; error 28 us; skew 500 ppm
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:52.638965 6777 init.cc:377] Logged in from keytab as kudu/127.16.90.252@KRBTEST.COM (short username kudu)
I20260926 14:43:52.639824 6777 webserver.cc:460] Webserver started at http://127.16.90.252:43819/ using document root <none> and password file <none>
I20260926 14:43:52.640333 6777 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:52.640393 6777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:52.640590 6777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:52.642539 6777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/data/instance:
uuid: "8b194aa82e2545cc9614987493c5dea3"
format_stamp: "Formatted at 2026-09-26 14:43:52 on dist-test-slave-x4sn"
server_key: "da4b191edd92f855ec185016d8cbdb98"
server_key_iv: "f03f6a780389693971b95b8796660a55"
server_key_version: "encryptionkey@0"
I20260926 14:43:52.643057 6777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/wal/instance:
uuid: "8b194aa82e2545cc9614987493c5dea3"
format_stamp: "Formatted at 2026-09-26 14:43:52 on dist-test-slave-x4sn"
server_key: "da4b191edd92f855ec185016d8cbdb98"
server_key_iv: "f03f6a780389693971b95b8796660a55"
server_key_version: "encryptionkey@0"
I20260926 14:43:52.646528 6777 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 14:43:52.648904 6793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:52.650069 6777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:43:52.650172 6777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/wal
uuid: "8b194aa82e2545cc9614987493c5dea3"
format_stamp: "Formatted at 2026-09-26 14:43:52 on dist-test-slave-x4sn"
server_key: "da4b191edd92f855ec185016d8cbdb98"
server_key_iv: "f03f6a780389693971b95b8796660a55"
server_key_version: "encryptionkey@0"
I20260926 14:43:52.650267 6777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:52.686013 6777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:52.689196 6777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:52.689376 6777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:52.695777 6777 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.252:46389
I20260926 14:43:52.696024 6845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.252:46389 every 8 connection(s)
I20260926 14:43:52.696753 6777 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
I20260926 14:43:52.699349 6846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:52.703049 6846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:43:52.706903 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 6777
I20260926 14:43:52.707037 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433562403030-16747-0/minicluster-data/master-2/wal/instance
I20260926 14:43:52.707294 16747 external_mini_cluster.cc:1428] Setting key f0613334f7b8d27fc6327a3cf2e1f1b2
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.252@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.252@KRBTEST.COM
I20260926 14:43:52.716907 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.705341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.16.90.254:43557)
0926 14:43:52.705474 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.705477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.705957 (+ 480us) server_negotiation.cc:407] Connection header received
0926 14:43:52.706865 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.706868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.706924 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.706996 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.708366 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.709058 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.710268 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.710512 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.710588 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.713292 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.713296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.713299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.713328 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.714811 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.715321 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.715325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.715327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.715402 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.715771 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.715775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.715776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.715956 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.716049 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.716628 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.716754 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:43:52.717718 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.705350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:43557 (local address 127.0.0.1:60106)
0926 14:43:52.705809 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.705852 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:52.706638 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:52.707198 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:52.707204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:52.707572 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:52.708209 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.708230 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.709182 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.709186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.710144 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.710154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.710651 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.710655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.710850 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.711486 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:52.711502 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:52.713140 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:52.714923 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.714932 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.714942 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.715233 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.715480 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.715483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.715485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.715625 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.716062 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:52.716068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:52.716346 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 14:43:52.716519 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 14:43:52.716806 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260926 14:43:52.717727 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.707905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.16.90.252:46389)
0926 14:43:52.708260 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.708268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.708289 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:43:52.708424 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.708429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.708618 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.708778 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.709476 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.710231 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.711258 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.711698 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.711780 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.714216 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.714223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.714235 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.714260 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.715617 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.716034 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.716040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.716041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.716120 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.716359 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.716361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.716362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.716587 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.716784 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.716969 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.717103 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":89,"threads_started":1}
I20260926 14:43:52.719100 6846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:43:52.721949 6771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } attempt: 3
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.252@KRBTEST.COM for kudu/127.16.90.253@KRBTEST.COM
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.253@KRBTEST.COM for kudu/127.16.90.252@KRBTEST.COM
I20260926 14:43:52.727743 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.719604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:32823 (local address 127.0.0.1:35536)
0926 14:43:52.719717 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.719730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:52.719796 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:52.720151 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:52.720154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:52.720278 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:52.720388 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.720395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.721202 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.721204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.721851 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.721861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.722333 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.722337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.722432 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.723110 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:52.723121 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:52.724557 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:52.726317 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.726320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.726322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.726501 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.726760 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.726762 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.726764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.726828 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.727222 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:52.727224 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:52.727308 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 14:43:52.727433 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 14:43:52.727548 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 14:43:52.727754 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.719733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.16.90.253:32823)
0926 14:43:52.719865 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.719870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.719882 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:43:52.719921 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.719924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.719967 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.720039 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.720507 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.721089 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.721965 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.722229 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.722321 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.724694 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.724698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.724701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.724740 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.726200 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.726590 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.726594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.726596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.726675 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.726911 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.726915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.726917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.727083 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.727182 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.727491 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.727587 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:43:52.728706 6846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:43:52.729821 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.720059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.16.90.254:43557)
0926 14:43:52.720165 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.720168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.720178 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:43:52.720205 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.720207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.720238 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.720296 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.720870 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.721456 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.722438 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.722741 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.722836 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.725455 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.725459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.725461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.725487 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.727457 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.728090 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.728093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.728095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.728176 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.728427 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.728430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.728431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.728653 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.728898 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.729560 (+ 662us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.729677 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 14:43:52.730602 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.722405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53740)
0926 14:43:52.722642 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.722655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:52.722730 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:52.722954 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:52.722957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:52.723165 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:52.723264 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.723271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.724137 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.724140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.724852 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.724863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.725387 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.725392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.725474 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.725779 (+ 305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:52.725790 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:52.727024 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:52.729192 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.729195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.729197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.729378 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.729677 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.729680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.729682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.729768 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.730154 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:52.730167 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:52.730251 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 14:43:52.730342 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 14:43:52.730462 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260926 14:43:52.731612 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.722619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53740 (local address 127.16.90.252:46389)
0926 14:43:52.722711 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.722714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.722726 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:43:52.722829 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.722831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.722870 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.722937 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.723359 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.723997 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.724986 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.725280 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.725351 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.727144 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.727147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.727150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.727171 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.729104 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.729483 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.729486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.729487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.729563 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.729873 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.729876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.729878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.730048 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.730157 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.731373 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.731474 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:43:52.732849 6696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } attempt: 3
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.252@KRBTEST.COM for kudu/127.16.90.252@KRBTEST.COM
I20260926 14:43:52.735625 6771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0: Bootstrap starting.
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.253@KRBTEST.COM
Sep 26 14:43:52 dist-test-slave-x4sn krb5kdc[6615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.252@KRBTEST.COM
I20260926 14:43:52.738865 6771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:52.738981 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.729172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53742)
0926 14:43:52.729312 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.729325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:52.729390 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:52.729785 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:52.729788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:52.729932 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:52.730005 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.730011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.731265 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.731268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.732006 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.732016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.732527 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.732530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.732598 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.734061 (+ 1463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:52.734072 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:52.735344 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:52.737390 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.737392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.737394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.737790 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.738041 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.738044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.738046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.738113 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.738532 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:52.738534 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:52.738607 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 14:43:52.738711 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 14:43:52.738827 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":1}
I20260926 14:43:52.739522 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.729208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.16.90.252:46389)
0926 14:43:52.729539 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.729548 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.729559 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:43:52.729586 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.729588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.729625 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.729696 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.730109 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.731165 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.732116 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.732434 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.732531 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.735466 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.735469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.735471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.735491 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.737289 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.737884 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.737886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.737888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.737962 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.738189 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.738192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.738193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.738418 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.738502 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.739269 (+ 767us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.739374 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"spinlock_wait_cycles":2304,"thread_start_us":120,"threads_started":1}
I20260926 14:43:52.739804 6771 log.cc:824] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:52.741637 6771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0: No bootstrap required, opened a new log
I20260926 14:43:52.742489 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.733209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:46389 (local address 127.0.0.1:53756)
0926 14:43:52.733364 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:52.733376 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:52.733431 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:52.734103 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:52.734106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:52.734288 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:52.734389 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.734397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.735407 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.735411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.736267 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:52.736280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.736893 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.736898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:52.737007 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.737421 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:52.737438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:52.738794 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:52.740796 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.740800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.740802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.741059 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.741400 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:52.741404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:52.741406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:52.741499 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:52.741968 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:52.741976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:52.742089 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 14:43:52.742184 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 14:43:52.742322 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 14:43:52.743146 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.733266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.16.90.252:46389)
0926 14:43:52.733828 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.733832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.733843 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:43:52.733872 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.733874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.733907 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.733973 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.734554 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.735262 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.736428 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.736748 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.737222 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.738969 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.738974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.738977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.739003 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.740657 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.741195 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.741198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.741200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.741282 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.741616 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.741619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.741620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.741825 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.742795 (+ 970us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.742874 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.742995 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"spinlock_wait_cycles":768}
I20260926 14:43:52.743317 6846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3: Bootstrap starting.
I20260926 14:43:52.745312 6771 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } }
I20260926 14:43:52.745545 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.731035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.16.90.253:32823)
0926 14:43:52.731127 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.731131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.731141 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:43:52.731166 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.731168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.731196 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.731254 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.732424 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.732882 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.733786 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.734012 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.734127 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.738064 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.738069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.738072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.738100 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.740753 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.744216 (+ 3463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.744220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.744221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.744305 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.744601 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.744604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.744605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.744817 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.744969 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.745307 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.745416 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:43:52.745533 6771 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c344e7fc9948269cbbe6a875695ad0, State: Initialized, Role: FOLLOWER
I20260926 14:43:52.745905 6771 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } }
I20260926 14:43:52.746114 6846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:52.746112 6696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9: Bootstrap starting.
I20260926 14:43:52.747300 6771 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:52.747337 6846 log.cc:824] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:52.748855 6696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:52.749696 6696 log.cc:824] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:52.749756 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:52.752388 6846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3: No bootstrap required, opened a new log
I20260926 14:43:52.751253 6670 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60120
W20260926 14:43:52.753571 6865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:43:52.753634 6865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:43:52.751135 6703 master_runner.cc:186] Error getting master registration for 127.16.90.254:43557: OK, instance_id { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" instance_seqno: 1790433832273278 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:43:52.754350 6703 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:43:52.749476 6853 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } } }
I20260926 14:43:52.754652 6853 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:52.755301 6696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9: No bootstrap required, opened a new log
I20260926 14:43:52.756767 6846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } }
I20260926 14:43:52.757040 6846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b194aa82e2545cc9614987493c5dea3, State: Initialized, Role: FOLLOWER
I20260926 14:43:52.757400 6846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } }
I20260926 14:43:52.757886 6868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } } }
I20260926 14:43:52.757869 6696 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } }
I20260926 14:43:52.758081 6696 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa843094f3f741e592ac3ed8ccb4f4a9, State: Initialized, Role: FOLLOWER
I20260926 14:43:52.758100 6868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:52.758368 6696 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } }
I20260926 14:43:52.758816 6846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:52.758764 6869 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } } }
I20260926 14:43:52.758953 6869 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:52.759433 6696 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:52.759810 6670 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 14:43:52.759923 6670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:52.761000 6670 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } }
I20260926 14:43:52.761933 6670 leader_election.cc:288] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [CANDIDATE]: Term 1 election: Requested vote from peers e4c344e7fc9948269cbbe6a875695ad0 (127.16.90.253:32823), 8b194aa82e2545cc9614987493c5dea3 (127.16.90.252:46389)
I20260926 14:43:52.762712 6745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e4c344e7fc9948269cbbe6a875695ad0"
I20260926 14:43:52.762952 6745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:52.763149 6820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b194aa82e2545cc9614987493c5dea3"
I20260926 14:43:52.763347 6820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:52.764750 6820 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa843094f3f741e592ac3ed8ccb4f4a9 in term 1.
I20260926 14:43:52.764942 6745 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa843094f3f741e592ac3ed8ccb4f4a9 in term 1.
I20260926 14:43:52.765466 6647 leader_election.cc:302] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa843094f3f741e592ac3ed8ccb4f4a9, e4c344e7fc9948269cbbe6a875695ad0; no voters:
I20260926 14:43:52.765973 6869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:52.766192 6869 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [term 1 LEADER]: Becoming Leader. State: Replica: aa843094f3f741e592ac3ed8ccb4f4a9, State: Running, Role: LEADER
I20260926 14:43:52.766489 6869 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } }
I20260926 14:43:52.764972 6777 master_runner.cc:186] Error getting master registration for 127.16.90.254:43557: OK, instance_id { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" instance_seqno: 1790433832273278 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:43:52.768227 6777 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 14:43:52.770064 6649 tablet.cc:2369] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:43:52.770887 6880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:43:52.770977 6880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:43:52.768489 6896 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa843094f3f741e592ac3ed8ccb4f4a9. Latest consensus state: current_term: 1 leader_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } } }
I20260926 14:43:52.772881 6896 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [sys.catalog]: This master's current role is: LEADER
W20260926 14:43:52.776324 6895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:43:52.776439 6895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:43:52.779888 6902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:52.780786 6902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:52.784466 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.772562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.16.90.252:46389)
0926 14:43:52.772785 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.772789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.775943 (+ 3154us) server_negotiation.cc:407] Connection header received
0926 14:43:52.776119 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.776122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.776167 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.776216 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.777110 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.777827 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.778862 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.779181 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.779251 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.781144 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.781148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.781150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.781175 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.782740 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.783185 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.783189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.783191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.783271 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.783545 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.783548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.783550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.783774 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.783865 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.784200 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.784312 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260926 14:43:52.789793 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.772237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.16.90.254:43557)
0926 14:43:52.772334 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.772338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.773665 (+ 1327us) server_negotiation.cc:407] Connection header received
0926 14:43:52.775093 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.775096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.775144 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.775209 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.777898 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.778650 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.782835 (+ 4185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.783150 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.783231 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.786445 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.786449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.786452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.786480 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.788007 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.788498 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.788502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.788504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.788585 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.788896 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.788900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.788902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.789105 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.789219 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.789537 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.789643 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:43:52.791543 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:52.771378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.16.90.253:32823)
0926 14:43:52.771490 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.771495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.773743 (+ 2248us) server_negotiation.cc:407] Connection header received
0926 14:43:52.774004 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.774008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.774058 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.774107 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.776189 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.776799 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.783357 (+ 6558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:52.783685 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:52.783758 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:52.787137 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:52.787357 (+ 220us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:52.787360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:52.787390 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:52.788966 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.789981 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.789985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.789986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.790065 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:52.790594 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:52.790597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:52.790599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:52.790806 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:52.790909 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:52.791296 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 14:43:52.791401 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:43:52.795050 6820 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [term 1 FOLLOWER]: Refusing update from remote peer aa843094f3f741e592ac3ed8ccb4f4a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:52.795068 6745 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [term 1 FOLLOWER]: Refusing update from remote peer aa843094f3f741e592ac3ed8ccb4f4a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:52.795885 6869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:52.796777 6869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:52.800511 6868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa843094f3f741e592ac3ed8ccb4f4a9. Latest consensus state: current_term: 1 leader_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } } }
I20260926 14:43:52.800627 6868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:52.801182 6853 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa843094f3f741e592ac3ed8ccb4f4a9. Latest consensus state: current_term: 1 leader_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } } }
I20260926 14:43:52.801299 6853 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:52.803860 6869 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } } }
I20260926 14:43:52.803970 6869 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:52.803961 6868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } } }
I20260926 14:43:52.804041 6868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:52.805851 6896 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } } }
I20260926 14:43:52.805950 6896 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:52.806705 6853 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa843094f3f741e592ac3ed8ccb4f4a9" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 43557 } } peers { permanent_uuid: "e4c344e7fc9948269cbbe6a875695ad0" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 32823 } } peers { permanent_uuid: "8b194aa82e2545cc9614987493c5dea3" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 46389 } } }
I20260926 14:43:52.806823 6853 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:52.807765 6905 mvcc.cc:205] Tried to move back new op lower bound from 7333616979121328128 to 7333616979014316032. Current Snapshot: MvccSnapshot[applied={T|T < 7333616979121328128}]
I20260926 14:43:52.812892 6902 catalog_manager.cc:1347] Generated new cluster ID: ce54ac6f335f4bcd98b64fa600d380e4
I20260926 14:43:52.812952 6902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:52.813403 6907 mvcc.cc:205] Tried to move back new op lower bound from 7333616979121328128 to 7333616979014316032. Current Snapshot: MvccSnapshot[applied={T|T < 7333616979121328128}]
I20260926 14:43:52.848060 6902 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:52.849460 6902 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:52.862152 6902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa843094f3f741e592ac3ed8ccb4f4a9: Generated new TSK 0
I20260926 14:43:52.863102 6902 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:43:52.883922 6850 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:52.881218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.16.90.252:46389)
0926 14:43:52.881341 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.881346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.882897 (+ 1551us) server_negotiation.cc:407] Connection header received
0926 14:43:52.883080 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.883084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.883131 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.883193 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.883794 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53778: BlockingRecv error: recv got EOF from 127.0.0.1:53778 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 14:43:52.885187 6700 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:52.881410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.16.90.254:43557)
0926 14:43:52.881512 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.881516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.884376 (+ 2860us) server_negotiation.cc:407] Connection header received
0926 14:43:52.884532 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.884536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.884582 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.884628 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.885064 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60136: BlockingRecv error: recv got EOF from 127.0.0.1:60136 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 14:43:52.886435 6776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:52.881805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.16.90.253:32823)
0926 14:43:52.881909 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.881913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.885620 (+ 3707us) server_negotiation.cc:407] Connection header received
0926 14:43:52.885774 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.885778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.885822 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.885869 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.886327 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35560: BlockingRecv error: recv got EOF from 127.0.0.1:35560 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 14:43:52.890508 6776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:52.889176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.16.90.253:32823)
0926 14:43:52.889266 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.889270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.889662 (+ 392us) server_negotiation.cc:407] Connection header received
0926 14:43:52.889858 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.889861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.889927 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.889973 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.890424 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35568: BlockingRecv error: recv got EOF from 127.0.0.1:35568 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 14:43:52.895062 6776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:52.893794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.16.90.253:32823)
0926 14:43:52.893881 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.893885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.894316 (+ 431us) server_negotiation.cc:407] Connection header received
0926 14:43:52.894471 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.894474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.894515 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.894559 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.894980 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35578: BlockingRecv error: recv got EOF from 127.0.0.1:35578 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 14:43:52.900101 6776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:52.898852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.16.90.253:32823)
0926 14:43:52.898936 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.898940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.899363 (+ 423us) server_negotiation.cc:407] Connection header received
0926 14:43:52.899511 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.899514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.899554 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.899596 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.900022 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35584: BlockingRecv error: recv got EOF from 127.0.0.1:35584 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 14:43:52.907397 6776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:52.906024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.16.90.253:32823)
0926 14:43:52.906134 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.906137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.906588 (+ 451us) server_negotiation.cc:407] Connection header received
0926 14:43:52.906761 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.906764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.906809 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.906855 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.907316 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35588: BlockingRecv error: recv got EOF from 127.0.0.1:35588 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 14:43:52.919358 6776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:52.917680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.16.90.253:32823)
0926 14:43:52.917794 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.917798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.918378 (+ 580us) server_negotiation.cc:407] Connection header received
0926 14:43:52.918574 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.918577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.918628 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.918679 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.919246 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35604: BlockingRecv error: recv got EOF from 127.0.0.1:35604 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 14:43:52.940352 6776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:52.938331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.16.90.253:32823)
0926 14:43:52.938460 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.938465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.939287 (+ 822us) server_negotiation.cc:407] Connection header received
0926 14:43:52.939501 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.939505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.939559 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.939614 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.940236 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35620: BlockingRecv error: recv got EOF from 127.0.0.1:35620 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 14:43:52.975601 6776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:52.974513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.16.90.253:32823)
0926 14:43:52.974684 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:52.974689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:52.974712 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:43:52.974868 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:52.974871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:52.974932 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:52.975041 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:52.975451 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35630: BlockingRecv error: recv got EOF from 127.0.0.1:35630 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260926 14:43:53.042344 6776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:53.041392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.16.90.253:32823)
0926 14:43:53.041520 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:53.041524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:53.041632 (+ 108us) server_negotiation.cc:407] Connection header received
0926 14:43:53.041767 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:53.041770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:53.041815 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:53.041890 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:53.042243 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35636: BlockingRecv error: recv got EOF from 127.0.0.1:35636 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 14:43:53.173295 6776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:53.172283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.16.90.253:32823)
0926 14:43:53.172445 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:53.172450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:53.172556 (+ 106us) server_negotiation.cc:407] Connection header received
0926 14:43:53.172681 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:53.172683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:53.172734 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:53.172803 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:53.173173 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35642: BlockingRecv error: recv got EOF from 127.0.0.1:35642 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 14:43:53.432334 6776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:53.431383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.16.90.253:32823)
0926 14:43:53.431520 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:53.431524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:53.431540 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:43:53.431725 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:53.431728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:53.431781 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:53.431863 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:53.432233 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35654: BlockingRecv error: recv got EOF from 127.0.0.1:35654 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:43:53.754930 6865 catalog_manager.cc:1259] Loaded cluster ID: ce54ac6f335f4bcd98b64fa600d380e4
I20260926 14:43:53.755012 6865 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0: loading cluster ID for follower catalog manager: success
I20260926 14:43:53.756749 6865 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0: acquiring CA information for follower catalog manager: success
I20260926 14:43:53.757871 6865 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4c344e7fc9948269cbbe6a875695ad0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:43:53.772199 6880 catalog_manager.cc:1259] Loaded cluster ID: ce54ac6f335f4bcd98b64fa600d380e4
I20260926 14:43:53.772264 6880 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3: loading cluster ID for follower catalog manager: success
I20260926 14:43:53.773867 6880 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3: acquiring CA information for follower catalog manager: success
I20260926 14:43:53.774976 6880 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b194aa82e2545cc9614987493c5dea3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:43:53.949960 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:53.946261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.16.90.253:32823)
0926 14:43:53.946507 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:53.946511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:53.946527 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:43:53.946639 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:53.946642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:53.946799 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:53.946891 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:53.947121 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:53.947733 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:53.948608 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:53.948835 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:53.948912 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:53.949148 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:53.949622 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:53.949712 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:53.949763 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 14:43:53.949810 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260926 14:43:53.956244 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:53.952773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.16.90.252:46389)
0926 14:43:53.952997 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:53.953001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:53.953015 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:43:53.953119 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:53.953121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:53.953237 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:53.953352 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:53.953547 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:53.954044 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:53.954864 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:53.955093 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:53.955176 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:53.955440 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:53.955876 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:53.955978 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:53.956038 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 14:43:53.956102 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":55,"threads_started":1}
I20260926 14:43:53.962926 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:53.958877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35676 (local address 127.16.90.253:32823)
0926 14:43:53.959126 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:53.959130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:53.959300 (+ 170us) server_negotiation.cc:407] Connection header received
0926 14:43:53.959401 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:53.959404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:53.959462 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:53.959518 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:53.959770 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:53.960551 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:53.961780 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:53.962075 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:53.962138 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:53.962468 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:53.962548 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:53.962628 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:53.962744 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 14:43:53.962798 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260926 14:43:53.963101 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:53.958979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.16.90.252:46389)
0926 14:43:53.959091 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:53.959096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:53.959551 (+ 455us) server_negotiation.cc:407] Connection header received
0926 14:43:53.959742 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:53.959746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:53.959793 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:53.959858 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:53.960086 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:53.960829 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:53.961947 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:53.962245 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:53.962330 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:53.962629 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:53.962696 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:53.962762 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:53.962909 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 14:43:53.962957 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:43:53.963349 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:53.958793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.16.90.254:43557)
0926 14:43:53.959041 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:53.959046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:53.959497 (+ 451us) server_negotiation.cc:407] Connection header received
0926 14:43:53.959597 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:53.959600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:53.959794 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:53.959892 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:53.960192 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:53.960757 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:53.961653 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:53.961893 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:53.961982 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:53.962378 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:53.962716 (+ 338us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:53.962791 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:53.963157 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 14:43:53.963220 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
I20260926 14:43:53.966141 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 6628
I20260926 14:43:53.971655 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 6703
I20260926 14:43:53.976528 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 6777
2026-09-26T14:43:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2912 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:43:54 dist-test-slave-x4sn krb5kdc[6993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:43:54 dist-test-slave-x4sn krb5kdc[6993](info): set up 2 sockets
Sep 26 14:43:54 dist-test-slave-x4sn krb5kdc[6993](info): commencing operation
krb5kdc: starting...
W20260926 14:43:54.929373 16747 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/test-user.keytab.
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.90.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
Entry for principal kudu/127.16.90.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab.
I20260926 14:43:55.048456 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:45209
--webserver_interface=127.16.90.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:45209,127.16.90.253:39399,127.16.90.252:40271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:43:55.152386 7006 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:43:55.152663 7006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:55.152696 7006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:55.152733 7006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:55.156653 7006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:55.156706 7006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:55.156723 7006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:55.156737 7006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:55.156757 7006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:55.162177 7006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:45209,127.16.90.253:39399,127.16.90.252:40271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.254.keytab
--principal=kudu/127.16.90.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.254:45209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.90.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:55.163201 7006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:55.164171 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:55.171844 7011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:55.171855 7012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:55.171895 7014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:55.172084 7006 server_base.cc:1034] running on GCE node
I20260926 14:43:55.172698 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:55.173707 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:55.174861 7006 hybrid_clock.cc:648] HybridClock initialized: now 1790433835174847 us; error 27 us; skew 500 ppm
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:55.177336 7006 init.cc:377] Logged in from keytab as kudu/127.16.90.254@KRBTEST.COM (short username kudu)
I20260926 14:43:55.178140 7006 webserver.cc:460] Webserver started at http://127.16.90.254:45431/ using document root <none> and password file <none>
I20260926 14:43:55.178630 7006 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:55.178694 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:55.178880 7006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:55.180843 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/data/instance:
uuid: "4200946eb4e145debb13cb9909a15d2e"
format_stamp: "Formatted at 2026-09-26 14:43:55 on dist-test-slave-x4sn"
server_key: "27af8f213169091ce79ee5a6b8260893"
server_key_iv: "fa00887385d2a9803feacefbfb541bb5"
server_key_version: "encryptionkey@0"
I20260926 14:43:55.181361 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/wal/instance:
uuid: "4200946eb4e145debb13cb9909a15d2e"
format_stamp: "Formatted at 2026-09-26 14:43:55 on dist-test-slave-x4sn"
server_key: "27af8f213169091ce79ee5a6b8260893"
server_key_iv: "fa00887385d2a9803feacefbfb541bb5"
server_key_version: "encryptionkey@0"
I20260926 14:43:55.184944 7006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:43:55.187155 7021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:55.188357 7006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:55.188463 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/wal
uuid: "4200946eb4e145debb13cb9909a15d2e"
format_stamp: "Formatted at 2026-09-26 14:43:55 on dist-test-slave-x4sn"
server_key: "27af8f213169091ce79ee5a6b8260893"
server_key_iv: "fa00887385d2a9803feacefbfb541bb5"
server_key_version: "encryptionkey@0"
I20260926 14:43:55.188550 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:55.211766 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:55.214568 7006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:55.214743 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:55.221403 7006 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.254:45209
I20260926 14:43:55.221663 7073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.254:45209 every 8 connection(s)
I20260926 14:43:55.222313 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/data/info.pb
I20260926 14:43:55.223351 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 7006
I20260926 14:43:55.223461 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-0/wal/instance
I20260926 14:43:55.223768 16747 external_mini_cluster.cc:1428] Setting key 0d85a50b1b432336cdb4cf8c920c22b9
I20260926 14:43:55.224957 7074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:55.228473 7074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.90.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.253@KRBTEST.COM" created.
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:55.242488 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.230583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.16.90.254:45209)
0926 14:43:55.230971 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.230986 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.231204 (+ 218us) server_negotiation.cc:407] Connection header received
0926 14:43:55.232238 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.232242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.232438 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.232604 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.233350 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.234388 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.235470 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.236058 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.236287 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.238371 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.238378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.238390 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.238416 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.240000 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.240410 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.240414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.240415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.240512 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.240738 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.240741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.240742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.241062 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.241157 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.241529 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.241753 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":277,"spinlock_wait_cycles":17280,"thread_start_us":132,"threads_started":1}
I20260926 14:43:55.242486 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.230583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:45209 (local address 127.0.0.1:37636)
0926 14:43:55.231011 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.231095 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:55.231981 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:55.232638 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:55.232644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:55.232966 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:55.233205 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.233217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.234514 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.234518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.235348 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.235357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.236202 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.236205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.236357 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.236897 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:55.236912 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:55.238232 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:55.240093 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.240101 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.240108 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.240334 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.240577 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.240580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.240582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.240677 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.241149 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:55.241154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:55.241298 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 14:43:55.241432 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 14:43:55.241687 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":42,"spinlock_wait_cycles":6912,"thread_start_us":96,"threads_started":1}
I20260926 14:43:55.244612 7074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:43:55.245254 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.245024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:39399 (local address 127.0.0.1:48806)
0926 14:43:55.245134 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.245205 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:39399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 14:43:55.245409 7025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.90.253:39399: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.253.keytab.
Entry for principal kudu/127.16.90.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.253.keytab.
I20260926 14:43:55.246160 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.245923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:39399 (local address 127.0.0.1:48812)
0926 14:43:55.246020 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.246068 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:39399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 14:43:55.246315 7074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.253:39399: Network error: Client connection negotiation failed: client connection to 127.16.90.253:39399: connect: Connection refused (error 111)
I20260926 14:43:55.246814 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.253:39399
--webserver_interface=127.16.90.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.253.keytab
--principal=kudu/127.16.90.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:45209,127.16.90.253:39399,127.16.90.252:40271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:43:55.295558 7074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } attempt: 1
I20260926 14:43:55.296424 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.296105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:39399 (local address 127.0.0.1:48818)
0926 14:43:55.296261 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.296334 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:39399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 14:43:55.297127 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.296910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:39399 (local address 127.0.0.1:48820)
0926 14:43:55.297002 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.297051 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:39399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 14:43:55.297243 7074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.253:39399: Network error: Client connection negotiation failed: client connection to 127.16.90.253:39399: connect: Connection refused (error 111)
W20260926 14:43:55.354630 7081 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:43:55.354997 7081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:55.355041 7081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:55.355100 7081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:55.359230 7081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:55.359289 7081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:55.359308 7081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:55.359323 7081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:55.359350 7081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:55.364607 7081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:45209,127.16.90.253:39399,127.16.90.252:40271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.253.keytab
--principal=kudu/127.16.90.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.253:39399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.90.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:55.365389 7074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } attempt: 2
I20260926 14:43:55.365756 7081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:55.366206 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.365886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:39399 (local address 127.0.0.1:48834)
0926 14:43:55.366041 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.366121 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:39399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 14:43:55.366874 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.366657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:39399 (local address 127.0.0.1:48838)
0926 14:43:55.366757 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.366808 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.253:39399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:43:55.366986 7074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.253:39399: Network error: Client connection negotiation failed: client connection to 127.16.90.253:39399: connect: Connection refused (error 111)
I20260926 14:43:55.367015 7081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:43:55.374876 7089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:55.375057 7081 server_base.cc:1034] running on GCE node
W20260926 14:43:55.374881 7086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:55.374897 7087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:55.375723 7081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:55.376770 7081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:55.377918 7081 hybrid_clock.cc:648] HybridClock initialized: now 1790433835377901 us; error 28 us; skew 500 ppm
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:55.380450 7081 init.cc:377] Logged in from keytab as kudu/127.16.90.253@KRBTEST.COM (short username kudu)
I20260926 14:43:55.381326 7081 webserver.cc:460] Webserver started at http://127.16.90.253:38089/ using document root <none> and password file <none>
I20260926 14:43:55.381851 7081 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:55.381915 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:55.382095 7081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:55.384083 7081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/data/instance:
uuid: "ec6ef908210b4340b9f766c8df4be26d"
format_stamp: "Formatted at 2026-09-26 14:43:55 on dist-test-slave-x4sn"
server_key: "eb2742d98f467b5a5ca3f9037e74f3c6"
server_key_iv: "c61b44e255280632e9acc0e1e2ae22f9"
server_key_version: "encryptionkey@0"
I20260926 14:43:55.384606 7081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/wal/instance:
uuid: "ec6ef908210b4340b9f766c8df4be26d"
format_stamp: "Formatted at 2026-09-26 14:43:55 on dist-test-slave-x4sn"
server_key: "eb2742d98f467b5a5ca3f9037e74f3c6"
server_key_iv: "c61b44e255280632e9acc0e1e2ae22f9"
server_key_version: "encryptionkey@0"
I20260926 14:43:55.388067 7081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:43:55.390239 7096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:55.391392 7081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:43:55.391502 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/wal
uuid: "ec6ef908210b4340b9f766c8df4be26d"
format_stamp: "Formatted at 2026-09-26 14:43:55 on dist-test-slave-x4sn"
server_key: "eb2742d98f467b5a5ca3f9037e74f3c6"
server_key_iv: "c61b44e255280632e9acc0e1e2ae22f9"
server_key_version: "encryptionkey@0"
I20260926 14:43:55.391601 7081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:55.405673 7081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:55.408543 7081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:55.408715 7081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:55.415275 7081 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.253:39399
I20260926 14:43:55.415515 7148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.253:39399 every 8 connection(s)
I20260926 14:43:55.416235 7081 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/data/info.pb
I20260926 14:43:55.418876 7149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:55.421986 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 7081
I20260926 14:43:55.422093 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-1/wal/instance
I20260926 14:43:55.422344 16747 external_mini_cluster.cc:1428] Setting key c10d68f3a56c51707689d329545ed9ec
I20260926 14:43:55.422508 7149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.90.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.90.252@KRBTEST.COM" created.
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.253@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:55.436148 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.424370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.16.90.254:45209)
0926 14:43:55.424487 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.424490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.425467 (+ 977us) server_negotiation.cc:407] Connection header received
0926 14:43:55.426354 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.426357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.426416 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.426488 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.427600 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.428175 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.429284 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.429551 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.429646 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.432779 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.432782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.432784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.432809 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.434091 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.434588 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.434592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.434594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.434689 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.434976 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.434979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.434981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.435205 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.435310 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.435835 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.435965 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:43:55.436858 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.424872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:45209 (local address 127.0.0.1:37650)
0926 14:43:55.425319 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.425360 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:55.426155 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:55.426652 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:55.426669 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:55.427004 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:55.427455 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.427468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.428296 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.428298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.429165 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.429174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.429700 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.429705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.429868 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.430386 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:55.430405 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:55.432620 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:55.434202 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.434207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.434218 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.434472 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.434776 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.434778 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.434779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.434898 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.435323 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:55.435329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:55.435561 (+ 232us) client_negotiation.cc:769] Sending connection context
0926 14:43:55.435759 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 14:43:55.436015 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
I20260926 14:43:55.438220 7149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.253@KRBTEST.COM for kudu/127.16.90.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.90.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.252.keytab.
Entry for principal kudu/127.16.90.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.252.keytab.
I20260926 14:43:55.446862 16747 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
/tmp/dist-test-tasks9vtzz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.252:40271
--webserver_interface=127.16.90.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.252.keytab
--principal=kudu/127.16.90.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.90.212:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.90.254:45209,127.16.90.253:39399,127.16.90.252:40271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:43:55.447152 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.438730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:39399 (local address 127.0.0.1:48850)
0926 14:43:55.438838 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.438851 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:55.438924 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:55.439522 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:55.439524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:55.439645 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:55.439767 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.439773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.440974 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.440977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.441545 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.441554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.442122 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.442125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.442294 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.442756 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:55.442768 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:55.443957 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:55.445641 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.445644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.445645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.445849 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.446112 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.446115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.446117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.446206 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.446720 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:55.446722 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:55.446804 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 14:43:55.446878 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 14:43:55.446984 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":51}
I20260926 14:43:55.447307 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.438792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.16.90.253:39399)
0926 14:43:55.439120 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.439132 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.439157 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:43:55.439199 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.439204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.439401 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.439531 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.439965 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.440811 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.441669 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.442029 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.442113 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.444070 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.444078 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.444089 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.444108 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.445555 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.445938 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.445945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.445947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.446029 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.446283 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.446286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.446288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.446614 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.446765 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.446985 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.447120 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":235,"thread_start_us":153,"threads_started":1}
I20260926 14:43:55.449345 7149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:43:55.449991 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.449770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35444)
0926 14:43:55.449873 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.449937 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:43:55.450174 7098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
I20260926 14:43:55.450906 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.450731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35446)
0926 14:43:55.450821 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.450860 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:43:55.451018 7149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:40271: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
I20260926 14:43:55.458184 7074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } attempt: 3
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.253@KRBTEST.COM
I20260926 14:43:55.466868 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.458787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:39399 (local address 127.0.0.1:48854)
0926 14:43:55.458970 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.458986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:55.459115 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:55.459357 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:55.459360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:55.459514 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:55.459713 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.459734 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.460778 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.460781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.461360 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.461370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.462028 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.462031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.462115 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.462579 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:55.462593 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:55.463935 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:55.465584 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.465586 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.465588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.465762 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.466045 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.466048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.466050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.466141 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.466498 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:55.466500 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:55.466575 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 14:43:55.466646 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:43:55.466748 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 14:43:55.466952 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.458809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.16.90.253:39399)
0926 14:43:55.458935 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.458944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.459045 (+ 101us) server_negotiation.cc:407] Connection header received
0926 14:43:55.459203 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.459206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.459254 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.459335 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.459844 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.460660 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.461458 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.461882 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.461988 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.464065 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.464069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.464072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.464098 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.465485 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.465841 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.465844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.465846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.465925 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.466225 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.466228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.466230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.466426 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.466512 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.466708 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.466818 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:43:55.468087 7074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:43:55.468708 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.468493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35462)
0926 14:43:55.468598 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.468641 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 14:43:55.469314 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.469095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35478)
0926 14:43:55.469223 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.469264 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 14:43:55.469429 7074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:40271: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
I20260926 14:43:55.500243 7149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } attempt: 1
I20260926 14:43:55.500571 7074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } attempt: 1
I20260926 14:43:55.501117 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.500798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35492)
0926 14:43:55.500961 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.501026 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 14:43:55.501257 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.500958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35506)
0926 14:43:55.501107 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.501171 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 14:43:55.501891 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.501648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35518)
0926 14:43:55.501772 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.501826 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 14:43:55.501992 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.501770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35534)
0926 14:43:55.501887 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.501924 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 14:43:55.502072 7149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:40271: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
W20260926 14:43:55.502097 7074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:40271: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
W20260926 14:43:55.558985 7155 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:43:55.559301 7155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:43:55.559324 7155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:43:55.559372 7155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:43:55.563416 7155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:43:55.563480 7155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:43:55.563498 7155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:43:55.563524 7155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:43:55.563551 7155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:43:55.568830 7155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.90.212:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.90.254:45209,127.16.90.253:39399,127.16.90.252:40271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/krb5kdc/kudu_127.16.90.252.keytab
--principal=kudu/127.16.90.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.90.252:40271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.90.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:43:55.569871 7155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:43:55.570225 7149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } attempt: 2
I20260926 14:43:55.571043 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.570725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35550)
0926 14:43:55.570881 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.570953 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 14:43:55.571130 7155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:43:55.572163 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.571933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35556)
0926 14:43:55.572043 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.572097 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 14:43:55.572279 7149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:40271: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
I20260926 14:43:55.577226 7074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } attempt: 2
I20260926 14:43:55.577970 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.577689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35560)
0926 14:43:55.577834 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.577895 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 14:43:55.578790 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.578567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35566)
0926 14:43:55.578677 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.578727 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:43:55.579906 7162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:55.579897 7164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:43:55.580106 7074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.90.252:40271: Network error: Client connection negotiation failed: client connection to 127.16.90.252:40271: connect: Connection refused (error 111)
W20260926 14:43:55.580310 7161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:43:55.580546 7155 server_base.cc:1034] running on GCE node
I20260926 14:43:55.581023 7155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:43:55.582139 7155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:43:55.583298 7155 hybrid_clock.cc:648] HybridClock initialized: now 1790433835583279 us; error 26 us; skew 500 ppm
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:43:55.585919 7155 init.cc:377] Logged in from keytab as kudu/127.16.90.252@KRBTEST.COM (short username kudu)
I20260926 14:43:55.586771 7155 webserver.cc:460] Webserver started at http://127.16.90.252:42479/ using document root <none> and password file <none>
I20260926 14:43:55.587277 7155 fs_manager.cc:359] Metadata directory not provided
I20260926 14:43:55.587334 7155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:43:55.587488 7155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:43:55.589668 7155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/data/instance:
uuid: "cdafffe6893047d3ad27f379c1896835"
format_stamp: "Formatted at 2026-09-26 14:43:55 on dist-test-slave-x4sn"
server_key: "fb54f03c61b72e2f61feaf9939beed77"
server_key_iv: "dd74e453dde8713f0e992a9e77abb0c0"
server_key_version: "encryptionkey@0"
I20260926 14:43:55.590165 7155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/wal/instance:
uuid: "cdafffe6893047d3ad27f379c1896835"
format_stamp: "Formatted at 2026-09-26 14:43:55 on dist-test-slave-x4sn"
server_key: "fb54f03c61b72e2f61feaf9939beed77"
server_key_iv: "dd74e453dde8713f0e992a9e77abb0c0"
server_key_version: "encryptionkey@0"
I20260926 14:43:55.593802 7155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 14:43:55.596251 7171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:43:55.597424 7155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:43:55.597555 7155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/data,/tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/wal
uuid: "cdafffe6893047d3ad27f379c1896835"
format_stamp: "Formatted at 2026-09-26 14:43:55 on dist-test-slave-x4sn"
server_key: "fb54f03c61b72e2f61feaf9939beed77"
server_key_iv: "dd74e453dde8713f0e992a9e77abb0c0"
server_key_version: "encryptionkey@0"
I20260926 14:43:55.597654 7155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:43:55.608397 7155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:43:55.611032 7155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:43:55.611191 7155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:43:55.617498 7155 rpc_server.cc:307] RPC server started. Bound to: 127.16.90.252:40271
I20260926 14:43:55.617733 7223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.90.252:40271 every 8 connection(s)
I20260926 14:43:55.618384 7155 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/data/info.pb
I20260926 14:43:55.620910 7224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:43:55.623193 16747 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu as pid 7155
I20260926 14:43:55.623325 16747 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9vtzz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433562403030-16747-0/minicluster-data/master-2/wal/instance
I20260926 14:43:55.623618 16747 external_mini_cluster.cc:1428] Setting key d17eda164b9d04054bd485b31394c75d
I20260926 14:43:55.624899 7224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.252@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.252@KRBTEST.COM
I20260926 14:43:55.635775 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.626481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.16.90.254:45209)
0926 14:43:55.626606 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.626610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.626876 (+ 266us) server_negotiation.cc:407] Connection header received
0926 14:43:55.627058 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.627060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.627101 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.627172 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.627849 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.628336 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.629327 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.629532 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.629615 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.631912 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.631915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.631918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.631946 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.633946 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.634394 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.634397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.634399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.634484 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.634796 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.634799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.634801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.635001 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.635091 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.635462 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.635572 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:43:55.636543 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.626358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.254:45209 (local address 127.0.0.1:37664)
0926 14:43:55.626761 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.626806 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:55.626956 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:55.627225 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:55.627231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:55.627543 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:55.627729 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.627740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.628444 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.628448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.629218 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.629227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.629665 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.629669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.629778 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.630350 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:55.630364 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:55.631740 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:55.634045 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.634054 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.634063 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.634320 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.634564 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.634567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.634569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.634717 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.635095 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:55.635099 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:55.635234 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 14:43:55.635390 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 14:43:55.635723 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260926 14:43:55.636545 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.624909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.16.90.252:40271)
0926 14:43:55.625279 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.625293 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.625326 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:43:55.625939 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.625954 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.626286 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.626721 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.627372 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.628338 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.629189 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.629536 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.629680 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.632316 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.632322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.632334 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.632356 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.633918 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.634394 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.634399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.634401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.634519 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.634774 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.634777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.634779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.635224 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.635355 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.635601 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.635922 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":246,"thread_start_us":117,"threads_started":1}
I20260926 14:43:55.637665 7224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.252@KRBTEST.COM for kudu/127.16.90.253@KRBTEST.COM
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.254@KRBTEST.COM
I20260926 14:43:55.645259 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.638072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.253:39399 (local address 127.0.0.1:48858)
0926 14:43:55.638337 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.638349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:55.638426 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:55.638757 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:55.638760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:55.638880 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:55.638962 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.638967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.639689 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.639693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.640202 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.640209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.640613 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.640617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.640705 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.641070 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:55.641090 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:55.642436 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:55.644009 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.644012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.644014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.644221 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.644475 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.644478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.644480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.644561 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.644876 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:55.644879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:55.644966 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 14:43:55.645045 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 14:43:55.645131 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260926 14:43:55.645390 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.638212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.16.90.253:39399)
0926 14:43:55.638349 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.638353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.638391 (+ 38us) server_negotiation.cc:407] Connection header received
0926 14:43:55.638520 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.638523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.638572 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.638642 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.639071 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.639568 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.640306 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.640522 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.640613 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.642561 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.642565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.642568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.642604 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.643913 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.644316 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.644320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.644322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.644395 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.644645 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.644647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.644648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.644798 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.644864 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.645146 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.645249 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:43:55.646109 7224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:43:55.646405 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.638665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.16.90.254:45209)
0926 14:43:55.638759 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.638762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.638803 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:43:55.638912 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.638914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.638949 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.639007 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.639516 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.640023 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.640708 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.640915 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.640992 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.643476 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.643479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.643481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.643501 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.644837 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.645229 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.645232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.645233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.645309 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.645590 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.645593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.645595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.645811 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.646007 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.646174 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.646271 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.252@KRBTEST.COM for kudu/127.16.90.252@KRBTEST.COM
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.90.253@KRBTEST.COM
I20260926 14:43:55.654202 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.646753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.16.90.252:40271)
0926 14:43:55.646858 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.646861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.646875 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:43:55.646968 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.646971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.647004 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.647067 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.647381 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.648096 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.649024 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.649315 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.649411 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.651241 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.651245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.651248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.651272 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.652610 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.653043 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.653046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.653048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.653132 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.653411 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.653413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.653414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.653600 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.653690 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.653950 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.654065 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":40}
I20260926 14:43:55.654206 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.646512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35580)
0926 14:43:55.646829 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.646838 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:55.646891 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:55.647085 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:55.647088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:55.647200 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:55.647279 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.647285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.648195 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.648198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.648916 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.648924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.649410 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.649413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.649468 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.649841 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:55.649850 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:55.651112 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:55.652724 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.652727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.652729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.652954 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.653239 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.653242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.653244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.653334 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.653700 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:55.653703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:55.653796 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 14:43:55.653898 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:43:55.654017 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260926 14:43:55.656124 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.647444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.16.90.253:39399)
0926 14:43:55.647551 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.647555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.647565 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:43:55.647655 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.647657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.647709 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.647774 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.648380 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.649061 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.649977 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.650309 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.650394 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.652777 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.652781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.652783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.652806 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.654518 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.654900 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.654903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.654904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.654987 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.655242 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.655246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.655247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.655403 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.655618 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.655792 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.655887 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:43:55.657457 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835: Bootstrap starting.
I20260926 14:43:55.659058 16747 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:43:55.659921 7224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:55.660640 7224 log.cc:824] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:55.660269 7048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4200946eb4e145debb13cb9909a15d2e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37676
I20260926 14:43:55.662467 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835: No bootstrap required, opened a new log
I20260926 14:43:55.663367 7149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } attempt: 3
I20260926 14:43:55.665833 7224 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } }
I20260926 14:43:55.666033 7224 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdafffe6893047d3ad27f379c1896835, State: Initialized, Role: FOLLOWER
I20260926 14:43:55.666363 7224 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } }
I20260926 14:43:55.666854 7230 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } } }
I20260926 14:43:55.666988 7230 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:55.667711 7224 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.253@KRBTEST.COM for kudu/127.16.90.252@KRBTEST.COM
W20260926 14:43:55.672482 7241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 14:43:55.672536 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.663749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35582)
0926 14:43:55.663848 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.663863 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:55.663935 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:55.664135 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:55.664138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:55.664311 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:55.664402 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.664408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.665334 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.665337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.666152 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.666161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.666678 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.666681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.666763 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.667230 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:55.667241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:55.668538 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:55.670341 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.670345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.670347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.670580 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.670857 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.670860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.670862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.670937 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.671909 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:55.671912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:55.672092 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 14:43:55.672236 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 14:43:55.672371 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 14:43:55.672554 7241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:43:55.673022 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.663813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.16.90.252:40271)
0926 14:43:55.663918 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.663922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.663935 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:43:55.664005 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.664008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.664050 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.664127 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.664498 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.665243 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.666258 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.666558 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.666671 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.668692 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.668696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.668698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.668721 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.670238 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.670686 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.670690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.670692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.670773 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.671048 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.671051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.671052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.671298 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.671381 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.672775 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.672883 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:43:55.672119 7155 master_runner.cc:186] Error getting master registration for 127.16.90.254:45209: OK, instance_id { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" instance_seqno: 1790433835219563 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:43:55.675598 7155 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:43:55.676033 7149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d: Bootstrap starting.
I20260926 14:43:55.678704 7149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:55.679257 7074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } attempt: 3
I20260926 14:43:55.679570 7149 log.cc:824] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:55.681718 7149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d: No bootstrap required, opened a new log
Sep 26 14:43:55 dist-test-slave-x4sn krb5kdc[6993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433835, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.90.254@KRBTEST.COM for kudu/127.16.90.252@KRBTEST.COM
I20260926 14:43:55.685750 7149 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } }
I20260926 14:43:55.686041 7149 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6ef908210b4340b9f766c8df4be26d, State: Initialized, Role: FOLLOWER
I20260926 14:43:55.686388 7149 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } }
I20260926 14:43:55.687075 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } } }
I20260926 14:43:55.687217 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:55.687808 7149 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:55.688060 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.679773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.16.90.252:40271)
0926 14:43:55.679937 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.679942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.679954 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:43:55.680057 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.680060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.680099 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.680161 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.680601 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.681376 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.682164 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.682496 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.682590 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.684750 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.684754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.684757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.684779 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.686223 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.686713 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.686717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.686718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.686801 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.687134 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.687137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.687139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.687352 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.687414 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.687819 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.687916 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 14:43:55.688060 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.679688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.90.252:40271 (local address 127.0.0.1:35592)
0926 14:43:55.679875 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 14:43:55.679889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:43:55.679975 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:43:55.680193 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:43:55.680197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:43:55.680385 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:43:55.680489 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.680496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.681470 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.681473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.682053 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:43:55.682062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.682584 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.682588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:43:55.682668 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.683142 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:43:55.683158 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:43:55.684627 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:43:55.686320 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.686323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.686326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.686563 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.686889 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:43:55.686892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:43:55.686894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:43:55.686985 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:43:55.687541 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:43:55.687544 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:43:55.687643 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 14:43:55.687757 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 14:43:55.687890 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 14:43:55.691344 7074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e: Bootstrap starting.
W20260926 14:43:55.693281 7255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:43:55.693346 7255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:43:55.693982 7074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e: Neither blocks nor log segments found. Creating new log.
I20260926 14:43:55.691089 7081 master_runner.cc:186] Error getting master registration for 127.16.90.254:45209: OK, instance_id { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" instance_seqno: 1790433835219563 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:43:55.694804 7081 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:43:55.695153 7074 log.cc:824] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:43:55.697244 7074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e: No bootstrap required, opened a new log
I20260926 14:43:55.700237 7074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } }
I20260926 14:43:55.700453 7074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4200946eb4e145debb13cb9909a15d2e, State: Initialized, Role: FOLLOWER
I20260926 14:43:55.700783 7074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } }
I20260926 14:43:55.701488 7258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } } }
I20260926 14:43:55.701828 7258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:55.702087 7074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:43:55.702152 7048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 14:43:55.702268 7048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:55.703505 7048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } }
I20260926 14:43:55.704509 7048 leader_election.cc:288] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [CANDIDATE]: Term 1 election: Requested vote from peers ec6ef908210b4340b9f766c8df4be26d (127.16.90.253:39399), cdafffe6893047d3ad27f379c1896835 (127.16.90.252:40271)
I20260926 14:43:55.706398 7123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4200946eb4e145debb13cb9909a15d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ec6ef908210b4340b9f766c8df4be26d"
I20260926 14:43:55.706645 7123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [term 0 FOLLOWER]: Advancing to term 1
W20260926 14:43:55.707057 7269 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:43:55.707118 7269 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:43:55.707913 7123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4200946eb4e145debb13cb9909a15d2e in term 1.
I20260926 14:43:55.709771 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4200946eb4e145debb13cb9909a15d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cdafffe6893047d3ad27f379c1896835"
I20260926 14:43:55.710011 7198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:43:55.711272 7198 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4200946eb4e145debb13cb9909a15d2e in term 1.
I20260926 14:43:55.711967 7023 leader_election.cc:302] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4200946eb4e145debb13cb9909a15d2e, cdafffe6893047d3ad27f379c1896835; no voters:
W20260926 14:43:55.716240 7027 tablet.cc:2369] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:43:55.717561 7006 master_runner.cc:263] No leader master found from master 4200946eb4e145debb13cb9909a15d2e
I20260926 14:43:55.717621 7006 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:43:55.717765 7258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:43:55.718020 7258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [term 1 LEADER]: Becoming Leader. State: Replica: 4200946eb4e145debb13cb9909a15d2e, State: Running, Role: LEADER
I20260926 14:43:55.718375 7258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } }
I20260926 14:43:55.720237 7258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4200946eb4e145debb13cb9909a15d2e. Latest consensus state: current_term: 1 leader_uuid: "4200946eb4e145debb13cb9909a15d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } } }
I20260926 14:43:55.720402 7258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:55.725775 7280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:43:55.730055 7280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:43:55.736565 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.712081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.16.90.252:40271)
0926 14:43:55.712191 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.712196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.723960 (+ 11764us) server_negotiation.cc:407] Connection header received
0926 14:43:55.724177 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.724180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.724232 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.724288 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.725301 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.726056 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.727188 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.727521 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.727599 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.732886 (+ 5287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.732891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.732894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.732923 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.734599 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.735118 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.735122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.735124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.735212 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.735524 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.735528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.735530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.735781 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.735886 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.736268 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.736390 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:43:55.739375 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.712654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48878 (local address 127.16.90.253:39399)
0926 14:43:55.712760 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.712764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.720853 (+ 8089us) server_negotiation.cc:407] Connection header received
0926 14:43:55.721169 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.721173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.721226 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.721277 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.723626 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.724417 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.730215 (+ 5798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.730570 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.730646 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.734901 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.734905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.734908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.734936 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.736420 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.737722 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.737725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.737727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.737805 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.738074 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.738077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.738079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.738274 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.738386 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.739107 (+ 721us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.739224 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:43:55.743981 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:55.712621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.16.90.254:45209)
0926 14:43:55.712914 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.712918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.722476 (+ 9558us) server_negotiation.cc:407] Connection header received
0926 14:43:55.722648 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.722651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.722695 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.722742 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.727777 (+ 5035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.728458 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.734006 (+ 5548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:55.734348 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:55.734417 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:55.738704 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:43:55.738707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:43:55.738709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:43:55.738735 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:43:55.740897 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.742647 (+ 1750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.742650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.742652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.742739 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:43:55.743008 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:43:55.743012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:43:55.743013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:43:55.743267 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:43:55.743383 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:55.743730 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 14:43:55.743841 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260926 14:43:55.751911 7198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [term 1 FOLLOWER]: Refusing update from remote peer 4200946eb4e145debb13cb9909a15d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:55.752658 7278 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:55.756551 7230 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4200946eb4e145debb13cb9909a15d2e. Latest consensus state: current_term: 1 leader_uuid: "4200946eb4e145debb13cb9909a15d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } } }
I20260926 14:43:55.756661 7230 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:55.759651 7123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [term 1 FOLLOWER]: Refusing update from remote peer 4200946eb4e145debb13cb9909a15d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:43:55.760250 7278 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:43:55.764146 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4200946eb4e145debb13cb9909a15d2e. Latest consensus state: current_term: 1 leader_uuid: "4200946eb4e145debb13cb9909a15d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } } }
I20260926 14:43:55.764266 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:55.772527 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4200946eb4e145debb13cb9909a15d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } } }
I20260926 14:43:55.772639 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:55.773252 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4200946eb4e145debb13cb9909a15d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } } }
I20260926 14:43:55.773368 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:55.773541 7281 mvcc.cc:205] Tried to move back new op lower bound from 7333616991220412416 to 7333616991106265088. Current Snapshot: MvccSnapshot[applied={T|T < 7333616991220412416}]
I20260926 14:43:55.776546 7258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4200946eb4e145debb13cb9909a15d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } } }
I20260926 14:43:55.776647 7258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e [sys.catalog]: This master's current role is: LEADER
I20260926 14:43:55.777693 7230 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4200946eb4e145debb13cb9909a15d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4200946eb4e145debb13cb9909a15d2e" member_type: VOTER last_known_addr { host: "127.16.90.254" port: 45209 } } peers { permanent_uuid: "ec6ef908210b4340b9f766c8df4be26d" member_type: VOTER last_known_addr { host: "127.16.90.253" port: 39399 } } peers { permanent_uuid: "cdafffe6893047d3ad27f379c1896835" member_type: VOTER last_known_addr { host: "127.16.90.252" port: 40271 } } }
I20260926 14:43:55.777792 7230 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:43:55.780090 7282 mvcc.cc:205] Tried to move back new op lower bound from 7333616991220412416 to 7333616991106265088. Current Snapshot: MvccSnapshot[applied={T|T < 7333616991220412416}]
I20260926 14:43:55.780236 7284 mvcc.cc:205] Tried to move back new op lower bound from 7333616991220412416 to 7333616991106265088. Current Snapshot: MvccSnapshot[applied={T|T < 7333616991220412416}]
I20260926 14:43:55.786996 7280 catalog_manager.cc:1347] Generated new cluster ID: f8d1a67495584b568dbaf9a21dde7e20
I20260926 14:43:55.787057 7280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:43:55.820568 7280 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:43:55.821878 7280 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:43:55.837391 7280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4200946eb4e145debb13cb9909a15d2e: Generated new TSK 0
I20260926 14:43:55.838313 7280 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:43:55.871611 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:55.868524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.16.90.253:39399)
0926 14:43:55.868651 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.868656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.870443 (+ 1787us) server_negotiation.cc:407] Connection header received
0926 14:43:55.870652 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.870655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.870712 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.870773 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.871443 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48888: BlockingRecv error: recv got EOF from 127.0.0.1:48888 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 14:43:55.873555 7078 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:55.869363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.16.90.254:45209)
0926 14:43:55.869494 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.869499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.872373 (+ 2874us) server_negotiation.cc:407] Connection header received
0926 14:43:55.872786 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.872790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.872843 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.872891 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.873419 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37704: BlockingRecv error: recv got EOF from 127.0.0.1:37704 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 14:43:55.874653 7227 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:55.870175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.16.90.252:40271)
0926 14:43:55.870283 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.870288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.872565 (+ 2277us) server_negotiation.cc:407] Connection header received
0926 14:43:55.872670 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.872674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.872733 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.872837 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.874539 (+ 1702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35622: BlockingRecv error: recv got EOF from 127.0.0.1:35622 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 14:43:55.878875 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:55.877420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.16.90.253:39399)
0926 14:43:55.877530 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.877534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.877968 (+ 434us) server_negotiation.cc:407] Connection header received
0926 14:43:55.878162 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.878165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.878250 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.878304 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.878776 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48900: BlockingRecv error: recv got EOF from 127.0.0.1:48900 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 14:43:55.883613 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:55.882308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.16.90.253:39399)
0926 14:43:55.882397 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.882401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.882829 (+ 428us) server_negotiation.cc:407] Connection header received
0926 14:43:55.882990 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.882993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.883039 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.883085 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.883520 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48908: BlockingRecv error: recv got EOF from 127.0.0.1:48908 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 14:43:55.888672 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:55.887593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.16.90.253:39399)
0926 14:43:55.887770 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.887774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.887885 (+ 111us) server_negotiation.cc:407] Connection header received
0926 14:43:55.888085 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.888088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.888137 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.888206 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.888578 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48922: BlockingRecv error: recv got EOF from 127.0.0.1:48922 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 14:43:55.895509 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:55.894571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48938 (local address 127.16.90.253:39399)
0926 14:43:55.894708 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.894711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.894775 (+ 64us) server_negotiation.cc:407] Connection header received
0926 14:43:55.894921 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.894925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.894977 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.895045 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.895415 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48938: BlockingRecv error: recv got EOF from 127.0.0.1:48938 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 14:43:55.906145 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:55.905207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.16.90.253:39399)
0926 14:43:55.905322 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.905327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.905436 (+ 109us) server_negotiation.cc:407] Connection header received
0926 14:43:55.905542 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.905545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.905599 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.905690 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.906039 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48950: BlockingRecv error: recv got EOF from 127.0.0.1:48950 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 14:43:55.925017 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:55.923900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.16.90.253:39399)
0926 14:43:55.924035 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.924040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.924196 (+ 156us) server_negotiation.cc:407] Connection header received
0926 14:43:55.924337 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.924340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.924389 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.924458 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.924891 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48958: BlockingRecv error: recv got EOF from 127.0.0.1:48958 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 14:43:55.960129 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:55.959026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48972 (local address 127.16.90.253:39399)
0926 14:43:55.959180 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:55.959184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:55.959271 (+ 87us) server_negotiation.cc:407] Connection header received
0926 14:43:55.959416 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:55.959420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:55.959467 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:55.959533 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:55.960002 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48972: BlockingRecv error: recv got EOF from 127.0.0.1:48972 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 14:43:56.027144 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:56.026050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.16.90.253:39399)
0926 14:43:56.026243 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:56.026247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:56.026313 (+ 66us) server_negotiation.cc:407] Connection header received
0926 14:43:56.026435 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:56.026437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:56.026489 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:56.026560 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:56.027021 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48988: BlockingRecv error: recv got EOF from 127.0.0.1:48988 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260926 14:43:56.158298 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:56.157279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.16.90.253:39399)
0926 14:43:56.157410 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:56.157414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:56.157487 (+ 73us) server_negotiation.cc:407] Connection header received
0926 14:43:56.157610 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:56.157612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:56.157661 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:56.157741 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:56.158200 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49000: BlockingRecv error: recv got EOF from 127.0.0.1:49000 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 14:43:56.417909 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:43:56.416885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.16.90.253:39399)
0926 14:43:56.417006 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:56.417011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:56.417110 (+ 99us) server_negotiation.cc:407] Connection header received
0926 14:43:56.417236 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:56.417239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:56.417290 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:56.417383 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:43:56.417803 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49016: BlockingRecv error: recv got EOF from 127.0.0.1:49016 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:43:56.673841 7241 catalog_manager.cc:1259] Loaded cluster ID: f8d1a67495584b568dbaf9a21dde7e20
I20260926 14:43:56.673930 7241 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835: loading cluster ID for follower catalog manager: success
I20260926 14:43:56.675535 7241 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835: acquiring CA information for follower catalog manager: success
I20260926 14:43:56.676718 7241 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cdafffe6893047d3ad27f379c1896835: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:43:56.694563 7255 catalog_manager.cc:1259] Loaded cluster ID: f8d1a67495584b568dbaf9a21dde7e20
I20260926 14:43:56.694638 7255 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d: loading cluster ID for follower catalog manager: success
I20260926 14:43:56.696311 7255 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d: acquiring CA information for follower catalog manager: success
I20260926 14:43:56.697417 7255 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec6ef908210b4340b9f766c8df4be26d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:43:56.935894 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:56.931973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.16.90.253:39399)
0926 14:43:56.932219 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:56.932224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:56.932240 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:43:56.932413 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:56.932416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:56.932540 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:56.932638 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:56.932918 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:56.933464 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:56.934380 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:56.934610 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:56.934699 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:56.935015 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:56.935470 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:56.935581 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:56.935660 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 14:43:56.935741 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":64,"threads_started":1}
I20260926 14:43:56.942348 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:56.938931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.16.90.252:40271)
0926 14:43:56.939164 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:56.939167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:56.939180 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:43:56.939292 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:56.939294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:56.939406 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:56.939497 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:56.939632 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:56.940165 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:56.941003 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:56.941221 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:56.941284 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:56.941535 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:56.941956 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:56.942042 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:56.942145 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 14:43:56.942201 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
I20260926 14:43:56.949220 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:56.945056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49020 (local address 127.16.90.253:39399)
0926 14:43:56.945169 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:56.945173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:56.945727 (+ 554us) server_negotiation.cc:407] Connection header received
0926 14:43:56.945816 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:56.945819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:56.945866 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:56.945933 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:56.946308 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:56.946804 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:56.948064 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:56.948375 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:56.948460 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:56.948765 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:56.948834 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:56.948901 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:56.949034 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 14:43:56.949090 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:43:56.949411 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:56.945088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.16.90.254:45209)
0926 14:43:56.945314 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:56.945318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:56.945643 (+ 325us) server_negotiation.cc:407] Connection header received
0926 14:43:56.945741 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:56.945744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:56.945893 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:56.945993 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:56.946336 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:56.947050 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:56.947909 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:56.948185 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:56.948278 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:56.948534 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:56.948826 (+ 292us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:56.948915 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:56.949208 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 14:43:56.949270 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260926 14:43:56.949425 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:43:56.945407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.16.90.252:40271)
0926 14:43:56.945511 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:43:56.945515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:43:56.945681 (+ 166us) server_negotiation.cc:407] Connection header received
0926 14:43:56.945822 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:43:56.945825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:43:56.945866 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:43:56.945921 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:43:56.946116 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:56.946779 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:56.948367 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:43:56.948672 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:43:56.948758 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:43:56.949030 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:43:56.949122 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:43:56.949191 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:43:56.949267 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 14:43:56.949316 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:43:56.955703 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 7006
I20260926 14:43:56.961059 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 7081
I20260926 14:43:56.965788 16747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9vtzz/build/debug/bin/kudu with pid 7155
2026-09-26T14:43:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2988 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5901 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (274537 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST